Case-Study: Security Operation Center Over Desentralized Network

Note: The experience shared here hold the key components from networking part as a glue to implement SOC tech over desentralized networks with technique such as “UDP Hole Punching” like the one used in P2P based on my personal deployment experience in one of company in Jakarta, Indonesia.

Part of the task for me to deliver this year is improving cybersecurity maturity level in one of subsidiaries of state-owned company group that — unfortunately, doesn’t have enough budget for Cybersecurity especially since their core business is not in cybersecurity (not even in IT). However, cybersecurity maturity improvement is mandatory and enforced from group level.

Let me re-phrase the words from one of the directors:

Imagine that we are a one big rich family. In the family, there are several children. Our parents are rich. Very rich. But as in reality, not all children fortunate enough to get the luxury of rich-family life. Consider that the children was given such an opportunity earlier, but somehow he / she messed up. So what’s left now? no more luxury, no more privilege, the children must live the life on its own.

Struggling in a business with huge debt is not uncommon. Lot of companies experience the similar situations. Cybersecurity is considered as luxury components in such business / company.

At certain point, I feel like it is a great opportunities. When the environment is full of luxury or privileges, and then you successfully improve cybersecurity level, that is good, but…not truly exceptional. Right? I mean, if you really really keep the art of engineering / security at the core of your hearts — and you are thinking that you are good enough, then ability to create an innovative solutions within such so unlucky environment is priceless. The satisfaction would be way over the joy instead of being success in a well supported environment.

Well, at least that’s what I thought.

So…, let’s move to one of the challenging problem I faced earlier, and how I formulated solutions along the way. It might not be the best solutions, but I think it is pretty cool due to its simplicity. I had a lot of fun exploring various tech before come to the final solution to implement. I think the similar situation could be faced by any other company, so hopefully this short introduction can be utilised by others as well.

Security Operation Center Integration

Improvement to cybersecurity maturity would require people, process, and technologies. There are many challenging issues to be solved, as in the title of this article, the one that I am going to discuss is related to technologies. More specifically, networking part.

Setting up Security Operation Center is part of the points in maturity improvement. In order to keep the budget low, I build the SOC by utilising any available components owned by the company. Here’s the funny fact: since the company is considered as enduser, naturally most of the responsibilities related to IT infrastructure handled by 3rd party (vendor). Starting with VM deployment and maintenance, VPN, firewall management, etc. That is good from vendor perspective, but not so good for the company. One change or modification to the IT environment / infrastructure would require the work from its vendor. If the contract is still running, then they will do it. If not, then the company must prepare the PO, or extend the previous contract.

Again, that is normal.

But since in this particular case budget is the most constrant, then solution must be formulated to achieve the goals without involving 3rd party support.

Here’s the design of SOC flow from theoretical concept based on the business requirement:

And here’s the implementation’s design concept I formulated based on above flows,

That is a one big and complex implementation, I know, but I am eager to move slowly to reach the final stage.

In the initial stage, I build a private cloud for the company to keep budget low. Paying for VMs managed by 3rd party or datacenter are expensive, so I utilised opensource technologies to tailor several unused PCs located in one of office branch connected to office’s wireless access point.

I couldn’t give the details of implementation for the sake of company’s confidentiality, but here’s the high level picture of how I implement them.

I was hired by the company initially to lead their digital telco businesses (so basically cybersecurity maturity improvement is the company additional task for me :p), one of the task was to create new product and business to increase company’s future profit revenues. The high level design above was one of the drafted solution (not the final one).

I adopt the implementation from one of the draft to hold cybersecurity services as in above red-squared lines.

Basically the idea is to tailor every compute power and storage spaces, build them into one cluster, that later on can hold virtualized operating system or containerized application for cybersecurity services as application on the top layer.

There is one big challenge in such above implementation: Networks.

We all knows that compute, storage, and network is part of cloud technologies. Networking part has been growing a lot with SDN (Software Defined Networking), so basically you can connect computing elements from different environment and locations without many complexity these days.

What are the use cases?

Let say I want to build the storage services to be utilised by virtualized OS using distributed tech like GlusterFS or Ceph, the storage could be available in different environment: 2 Synology Diskstation in HQ office, 3 baremetal servers (with 1TB each) in DC A, 2 backup server in DC B, 5 PC in branch office. Each of the components connected to different policy role and none of them have public IP. How to connect them into private cloud?

Another requirements, the staging, development, production servers to protect are located in different environment. Traditionally, SOC providers utilise VPN (Virtual Private Networks). That is easy if you have the SOC component controlled environment like having public IP or static private IP address, but if you have collection of standard PCs connected to office wireless (sometimes even unstable; so moving to different wireless AP), utilising traditional solution like VPNs come with several more challenges.

Networking is the glue to such “cheap”-er solutions.

In such situation, you’ll need to utilised decentralized networks.

I am a fans of old tech that empowering modern tech. Many people talks about sexy tech terminologies (so it sounds complex, cool, and expensive), while actually many of them was simply old tech that is wrapped / packaged humanly to be consumed by non-tech people. That is OK tho 😊

One of the cool movements I observed since couple of years back is about decentralized networks.

Did you know that the famous bitcoin was basically coming from philosophy of decentralizing monetary system? The tech junkies realized how monetary system was monopolized through government and central bank, so they come out with ideas of crypto currencies that is decentralized to liberate people.

If you are junkies enough about tech and loved watching geek serial / movies, you will realize Silicon Valley’s TV serial was basically trying to utilise decentralization solution to empower their product.

I would like to take one sample of this kind of movement project. Let’s read a little bit mission from redecentralize project,

Believe me, that is a strong message.

People who believe on decentralized networks produce various innovative tech. I think blockchain is the most interesting part of bitcoin in terms of tech. It was (considered) ‘nothing’ in terms of value many years ago, but when it goes mainstream, human realizes its anonymity feature can be utilized to empower their businesses (dirty one hehe), then suddenly it become valuable.

I put the 2 words in bold: feature, utilized.

This kind of movements actually produce many tech as we can see here,

GitHub — redecentralize/alternative-internet: A collection of interesting new networks and tech…

A collection of interesting networks and technology aiming at re-decentralizing the Internet. If you would like to help…

github.com

But in order to make them valuable, you need to find a suitable (and mostly profitable) use case to utilize them. I found that the alternative-internet can help low-budget business that unable afford expensive solution like public clouds.

I don’t says that I am against business, nor saying against anti-monopoly of internet technologies. I just utilised what’s available, explore / research what is possible as solution, and use them wisely to achieve the objective.

So, let’s return to our SOC use case before I am talking too much on philosophy stuff 😁

ZeroTier One

After exploring pretty much opensource (and alternate) solutions, combining different tools that take quite lot of time to research, I come to conclusion to use ZeroTier. Not only it is simple enough to use (I am not going to stay for long time in the company, so the solution must be simple enough to be handed over to the others), but it is suitable to the network environment policy where the company’s asset available: all the asset are basically OPEN to initiate connection, but restricted to receive initial connection.

I hope the reader are wise enough to read more details ZeroTier’s “Protocol Design Whitepaper” since I am not going to discuss in details here.

It contains all the tech details.

When I am saying that all assets are basically “OPEN to initiate connection, but restricted to receive initial connection”, that is a typical NAT-ed network environment. That’s why I chose ZeroTier for the current deployment.

ZeroTier give “magic” sense to non-tech people. For example, when I introduce to the team in the company, the first question for them: how is it possible VM installed on top of PCs located deep inside office networks with firewall and NAT policy, reachable easily via normal URL without modifying anything in the intermediate network? (modify something in the intermediate would need payment to the 3rd party as provider, remember?)

Here’s the visualization to understand the concept.

One of the “magic” utilised by ZeroTier is the one called “UDP Hole Punching”, it is borrowed from old P2P (Peer-to-Peer) technique.

From the diagram above, it is not possible for normal user located at home with private ip address 172.16.31.5 to reach SOC apps in the office with private ip address 192.168.0.3. However, both of them can reach any public IP in the internet through NAT (Network Address Translation) managed by the router.

For example, if there is a website with public IP address 8.8.8.8, here’s illustration of what happened whenever machine in SOC Apps would like to open that website:

SOC Apps tried to open the website public IP, when the initiation packet come to the router office, the router will add into its NAT table that whenever there is a communication between private ip:port 192.168.0.3:12345 and 8.8.8.8:80 then allow and translate the packet. Let’s not discuss about translation part as it will add another complexity that is out-of-the topic.

Next, what about if the destination is not website, but a Peer-to-Peer (P2P) application root servers to let people share files between two NAT’ed system?

P2P using technique called “Hole Punching”. Here’s excerpted text from Wikipedia about hole punching.

Hole punching (or sometimes punch-through) is a technique in computer networking for establishing a direct connection between two parties in which one or both are behind firewalls or behind routers that use network address translation (NAT). To punch a hole, each client connects to an unrestricted third-party server that temporarily stores external and internal address and port information for each client. The server then relays each client’s information to the other, and using that information each client tries to establish direct connection; as a result of the connections using valid port numbers, restrictive firewalls or routers accept and forward the incoming packets on each side.

ZeroTier uses UDP Hole Punching.

Most of the “magic” provided by ZeroTier available from the ZeroTier Network Hypervisor Core, you can read the code from github. The directory contains the real ZeroTier: a completely OS-independent global virtual Ethernet switch engine. This is where the magic happens.

ZeroTierOne/node at master · zerotier/ZeroTierOne

This directory contains the real ZeroTier: a completely OS-independent global virtual Ethernet switch engine. This is…

github.com

Let say, for UDP Hole Punching technique would require sending HELLO packet to the peer,

ZeroTier doesn’t stop after “punching the hole”. ZeroTier mission is to provide a smart Ethernet switch for planet Earth. That means ZeroTier can provide virtual IP address for each connected devices and let each device communicate directly over constructed desentralized network.

Let imagine that we are going to integrate devices from cellphone, PCs, synology disk, servers, etc located in different networks through ZeroTier then each device can have new private IP address routed through our own desentralized network.

ZeroTier solution also produce better performance compared to VPN especially if you can design the resource distribution wisely. For example, supposed that you are using a VPN then the traffic will be centralized, all connected devices will go to the VPN server first before routed to the destination. If the VPN server are located in different network, or even different countries then the latency would be increased. Since ZeroTier use peer-to-peer concept, then in case endpoint node are using particular ISP (i.e in Indonesia using Telkom IndiHome) and the SIEM also using same ISP, the packets would be routed internally inside IndiHome network (look at Hole Punching technique above). If you are using VPN methods, the packets most probably will go to different ISP so latency performance would be affected especially for big size of logs.

I hope by now, the reader have clearer insights to the blueprint / design I described earlier in this article where various compute and storage component are tailored together to compose private cloud through desentralized network with ZeroTier as the glue.

Is it safe? Doesn’t it mean the company opened its network to outsiders?

Well, there are a lot of discussion about this such as this one.

Google behind the concept of BeyondCorp that liberate their employee from utilising tools such as VPN to access their corporate resources. Tools similar to ZeroTier should be in-place, but not the only one.

Desentralized network is one thing, and could be only small set of the whole solution. There are various mechanism can be deployed to authorized user from accessing company application or asset such as Security Operation Center.

In terms of ZeroTier itself, even though we can’t discard their root servers, we can use one of our own server (with public IP address) to become controller node. At least to minimize the involvement of ZeroTier’s controller node to knows each connected devices into the constructed network.

Each ZeroTier node also incorporated crypto, so not all devices can join our private network.

Another question would be, what about if one of the node compromised and can access the whole node in the network? Well, I usually replied, “What about if one of your node in the VPN network get compromised?”, of course the attacker can get access to the rest of node connected to the VPN.

That’s why it is still mandatory to do further mitigation such as hardening all the node involved in the network as standard practice.

All in all, I hope the experience shared here can give benefit especially to small / big company with limited budget that can’t afford expensive equipment. Not only for Cybersecurity solution, but also as private cloud infrastructure as I noted a little bit in this article.

I think desentralization of internet would go mainstream in various aspect in the future especially with 5G network rolls-out. More and more devices to be connected, and speed are increased everywhere. As long as you know how to tailor them through the magic of networks, you can have pretty good computing power available to use.

The only things left is, use case, and innovative solution to utilise them.