OpenVMS x86-64 Guest Instances and Host Firewalls

Running OpenVMS x86-64 on a virtual machine requires careful attention to network connectivity between the OpenVMS instance(s) and the outside network as well as the virtual machine host.

Many hosts have integral firewalls. Default firewall settings can make seemingly obvious actions problematic. This is particularly true when local mobiles, desktops, or servers to host the underlying virtual machine. Local machines hosting virtual instances is particularly common in development, testing, and hobbyist environments.

It is unusual to configure an OpenVMS virtual instance without a network connection. Simulated serial interfaces are used to bootstrap the OpenVMS system. Once startup has completed, most communications, e.g., ssh and sftp, are done over the network connection. Once TCP/IP comes online, the OpenVMS instance appears to most local machines as just another system on the local area network. Host machine configuration settings and defaults affect connectivity. Choices and defaults have consequences.

Virtualized OpenVMS instances work as intended if one verifies that there are no firewall settings on the virtual machine host that interfere with IP connectivity. Microsoft Windows comes with an integral firewall, Microsoft Defender, whose default connection rules prohibit incoming ICMP messages. ICMP is the protocol used to support the PING and TRACEROUTE functions.

The other day, I provisioned a fresh OpenVMS virtual instance from scratch, following the instructions in the OpenVMS Installation Guide.¹ It has been about a year since I created an OpenVMS virtual instance from scratch. For the last year or so, I have recycled the virtual machine I had used for previous OpenVMS field test versions. The steps seem straightforward:

• create two VirtualBox hard drives per the instructions;
• unpacke the OpenVMS ISO-DVD image and attach it as a virtual DVD; and
• created a network interface as described in the OpenVMS Installation Instructions after reviewing the relevant chapter in the VirtualBox® User Manual.²

Bootstrap the virtual machine from the ISO-DVD image and perform a standard OpenVMS installation.

Once the OpenVMS system was running, configure TCP/IP services. In the summary below, the IP addresses are:

• VirtualBox host: 10.0.0.19
• Virtual OpenVMS instance: 10.0.0.20
• VirtualBox host: 10.0.0.19
• Uninvolved Windows Professional workstation: 10.0.0.18
• Network-attached printer: 10.0.0.10
• Gateway: 10.0.0.1

Testing network connectivity:

• [On OpenVMS] TCPIP PING 10.0.0.1
• [On OpenVMS] TCPIP PING 10.0.0.10
• [On OpenVMS] TCPIP PING xx.xx.xx.xx
• [On OpenVMS] TCPIP PING 10.0.0.19
• [On OpenVMS] TCPIP PING 10.0.0.18
• [On VirtualBox host] SFTP session on OpenVMS instance
• [On VirtualBox host] PING 10.0.0.20
• [On other Windows Professional system] PING 10.0.0.20

The failure to PING the Windows systems was disquiting. After some thought, it became clear that Microsoft Defender could be the source of the problem. Microsoft Defender includes a local firewall. The default configuration for Microsoft Defender disables Incoming ICMP. To make it particularly unobvious, Microsoft labeled the relevant rule as part of Microsoft File and Printer Sharing, Figure 1.

Figure 1 Microsoft Defender Inbound Filtering Rules

The local firewall blocking ICMP is an easily resolvable annoyance. However, it illuminates a more fundamental issue.

An OpenVMS x86-64 guest instance on a virtual machine is both simpler and more complex than running on physical hardware. Many hardware issues are dealt with by the virtual machine monitor. However, the underlying virtual machine environment must still be maintained and managed. Many will be managing both their OpenVMS environment and the underlying virtual machine monitor. Running OpenVMS on a personal system or workstation most often implies serving in both roles.

In the case at hand, the Microsoft Defender default settings allowed both ssh and sftp access to the OpenVMS instance from the host system, but did not allow ICMP access from the virtual instance to the host system. This would have escaped notice had I not used OpenVMS PING to check connectivity. A similar unconsidered default protection rule on the hosting system's firewall could have more significant impact had it gone unnoticed.

The larger lesson is that one needs to consider both the host system and the virtual instance settings when verifying virtual system functionality and troubleshooting problems.

This is especially true when running one or more virtualized OpenVMS instances as part of an OpenVMScluster. A single x86-64 system may be hosting one or more virtualized OpenVMScluster members.

Notes

[1]	VSI (2023, April) VSI OpenVMS x86-64 V 9.2-1 Installation Guide, Chapter 2
[2]	Oracle (2023, April 18) Oracle® VirtualBox® User Manual Version 7.0.8, Chapter 7

References

URLs for referencing this entry

• Home
• Services
  • OpenVMS
  • Windows
  • Printers
  • PostScript
  • Architectures
  • Networks
  • Embedded Systems
• Speaking
• Upcoming Events
• TechnicalCounsel^SM News Archive
• Ruminations – An IT Blog 
• The OpenVMS Consultant 
• Presentations
• Publications
• Technology Demonstrations
• Bookstore
• Contact Us

http://www.rlgsc.com

+1 (718) 463 1079