There was a time, not too long ago, that I used to dream of running a website that had HTTPS written in the address bar. Most big websites had it, but very few little ones and personal blogs did. I couldn’t because it required a lot of money (for a student, at least). Today, thanks to the efforts by Let’s Encrypt, Cloudflare and many other organizations which gave away basic SSL certificates for free, all of the domains I own are on HTTPS. But that’s not even the best part.
The best part is that HTTPS is now seen less as a luxury by small website owners and more of a necessity. Part of the reason for this is Google’s penalty in terms of search rankings for websites without HTTPS. The other part, and this is more important for the little non-business blog owners who do not care much about traffic and SEO, Google Chrome (>= 68) has started displaying a Not Secure for non-HTTPS websites. I would say that Google is doing a great job at providing people an incentive to switch. I hope Mozilla Firefox follows in Chrome’s footsteps in this regard.
non-https website on Google Chrome
non-https website on Mozilla Firefox
On similar lines, Github now only shows the entire web address (along with the protocol) if it is on HTTP. If the link posted is HTTPS, Github will truncate the protocol part and only show the domain name.
For example,
https://example.com
will be rendered as
https://example.com
while
https://example.com
(notice https) will be rendered as
example.com
If you notice, this is opposite of what happens in browsers today, http is truncated and https is emphasized.
I personally hope more and more website and browsers treat HTTPS as the default and HTTP as the exception, and not the other way round. The merits of using HTTP over HTTPS are either obsolete or negligible (read more about HTTPS here: ELI5 – How HTTPS Works). On the other hand, if done properly, HTTPS can be much faster than HTTP.
We already know that the more people using encryption, greater will be the overall value of using it. I’ll end here with a quote by Bruce Schneier, a cryptology badas- expert.
Encryption should be enabled for everything by default, not a feature you turn on only if you’re doing something you consider worth protecting.
This is important. If we only use encryption when we’re working with important data, then encryption signals that data’s importance. If only dissidents use encryption in a country, that country’s authorities have an easy way of identifying them. But if everyone uses it all of the time, encryption ceases to be a signal. No one can distinguish simple chatting from deeply private conversation. The government can’t tell the dissidents from the rest of the population. Every time you use encryption, you’re protecting someone who needs to use it to stay alive.
New cloud setup. YAAY! Self hosted, encrypted and scalable. Plus comes with a nice web interface, native Linux and Android clients and its very own app store. I’ll first write about the setup itself, and then some of my personal thoughts over the entire private cloud exercise.
Features Overview
The major components of the setup include the following
NextCloud 11 on Ubuntu using Digital Ocean’s one click installer on a 5 USD cloud vps
NextCloud sync client for Arch and Android on desktop and phone respectively for data sync
DavDroid for contacts and calender sync on Android (uses WebDAV)
Optional redundant backup and client side encryption using GnuPG (see below)
Pros Vs Cons
So I now have a proper private cloud, self hosted, synced across mobile and desktop (including contacts, messages and calender), optional client-side encryption and scalable (♥DigitalOcean♥). What’s amazing is that I never had a native Google Drive client on desktop, but now I have a native NextCloud client, and it just works. And yes, it isn’t all sunshine and rainbow. There are some serious trade-offs which I should mention at this point, to make this fair.
No Google Peering, hence backing up media is going to be a struggle on slow connections
Google’s cloud is without a doubt more securely managed and reliable than my vps.
Integration with Android is not as seamless as it was with Google apps, sync is almost always delayed (By 10 minutes. Yes, I’m an impatient (read ‘spoiled’) Google user)
Server maintenance is now my responsibility. Not a huge deal, but just something to keep in mind
Having said that, most of it is just a matter of getting familiar with the new set of tools in the arsenal. I’ve tried to keep most things minimal. Using few widely adopted technologies and keeping them regularly updated, sticking to the best practices and disabling any unwanted, potentially dangerous defaults and with that the server is secure from most adversaries. Let’s first define what “secure” means in the current context using a threat model.
Threat Model
The only thing worse than no security, is a false sense of security
Instead of securing everything in an ad hoc fashion, I’m using this explicitly defined threat model, which will help me prioritize what assets to secure and the degree of security, and more importantly, what threats I’m NOT secure against.
Compromised end device (Laptop): Since data is present unencrypted on my end, an adversary having access to my computer via say a ssh backdoor can easily get access to all of my (unencrypted) data. Private keys cannot be compromised as they are password protected. A keylogger might be able to sniff out my password which can then be used to decrypt any encrypted data.
Compromised end device (Mobile phone): Since data cannot be decrypted on the mobile, all encrypted data would remain secure. Only the unencrypted files will get compromised. However, if an adversary gets access to my unlocked cell phone, securing cloud data would be the least of my worries.
Man In The Middle (MITM): As long as Let’s Encrypt does it’s job, TLS used should be enough to secure the data against most adversaries eavesdropping on my network. It would not protect me if Let’s Encrypt (or any other CA) gets compromised and an adversary makes duplicate certificates against my domain and uses it to eavesdrop the traffic, the possibility of which is rare.
Server Compromise: If the server is compromised through any server side vulnerability (assume root access) and an attacker gets access to everything on the server, all unencrypted files are compromised, which would include contacts/calender lists. Since the decryption key is never transmitted to the server, encrypted files won’t be compromised.
Why Client Side Encryption
The entire exercise would look pretty pointless if I just took all my data from G Drive and pushed it to NextCloud. And from the previous cloud server attempt, I know how uncomfortable it is to have your data accessible from the network all the time. Those reasons were more than enough for me to go for an encrypted cloud solution. Although it would still look pointless if you were to ask me why didn’t I just encrypt the data and upload it to G Drive again. The answer is simply because I didn’t want to.
After some research (being a novice with security, that was a must), I came up with a list of guidelines that I had to write my solution on.
Use of symmetric key cryptography for file encryption, particularly AES-128
Memorizing the AES key or using public key cryptography to store the key of file en/decryption on disk. (Not sure which is the proper way of doing it, although I’ve asked the experts for help)
Encryption
There are a lot of tools one can use for data encryption. I used Gnu’s Privacy Guard (GnuPG or simply GPG). It is anything but easy to use. But the nice part is that it just works, is extensively reviewed by experts and has been around since I was 4 years old. So in theory,
Generate a strong passphrase for the encryption, and encrypt it using the public key you just generated. Store it locally someplace secure
Get a list of all files and directories from a specific folder using find (for one time backups), or use rsync with a local sync copy (for incremental backups)
Iterate the list (of all or changed files). If item is a directory, create that directory, if item is a file, encrypt the file and push it to that directory.
After encryption, you’re left with either two or three directories, /original-dir, /remote-encrypted and optionally, /local-unencrypted-sync
The additional (local sync) directory is useful when incremental backups are required and rsync uses this directory to keep track of changes, and only (re)encrypts those files that have been added/changed since last sync. Useful to setup a cron job. At this point, you can delete the files in your /original-dir safely
Decryption is just the opposite of this. You supply the location of your /remote-encrypted directory and the script generates a new directory with unencrypted content.
Original directory
Encrypted backup directory
This does the job for now. Here’s the script that I’m currently using. I wanted to enable sync without the need for a helper directory, just like Git does (it stores the changes in the same directory in a .git/ directory). Will update it if I manage to get that done.
In Closing
Eighteen months ago, I wrote on how to create a ‘cloud’ storage solution with the Raspberry Pi and half a terabyte hard disk that I had with me. Although it worked well (now that I think about it, it wasn’t really a cloud. Just storage attached to a computer accessible over the network. Wait, isn’t that a cloud? Damn these terms.), I was reluctant to keep my primary backup disk connected to the network all the time, powered by the tiny Pi, and hence I didn’t use it as much I had expected. So what I did then was what any sane person would’ve anyway done in the first place, connect the disk with a usb cable to the computer for file transfers and backups.
Earlier this year, I switched ISPs and got this new thing called Google Peering, which enabled me to efficiently backup all my data to the real ‘cloud’ (Google Drive). That worked, and it was effortless and maintenance free. And although Google doesn’t have a native Linux client yet, the web client was good enough for most things.
And that was the hardest thing to let go. Sync and automatic backups were, for me, the most useful feature of having Google around. And while everything else was easy to replace, the convenience of Drive is something that I’m still looking for in other open source solutions, something I even mentioned in my previous post on privacy.
So although I now have this good enough cloud solution, it definitely isn’t for everyone. The logical solution for most people (and me) would be to encrypt the data and back it up to Google Drive, Dropbox or others. I haven’t tried, but Mega.nz gives 50GB of free tier end to end encrypted storage. Ultimately, it makes much more sense to use a third party provider than doing it all yourself, but then again, where’s the fun in that! Thank you for reading.
In spite of my inclination towards cyber security from an early age (relative to when I ‘discovered’ the Internet), I never was a big fan of privacy over the web. I knew some bits here and there about it, like how my data is used to serve me targeted content, how tracking happens even after I close the browser tabs and how companies watch me visiting sites and track my habits. Heck, I found it fascinating that I saw adverts from third party companies about the products that I was currently researching about. Internet, to me, was like a close friend who knows everything about you, your habits and interests, your lifestyle and more. And when I say friend, it isn’t metaphorical. I literally trusted the web for every bit of work-life thing that I got involved into. I liked that my email was always synced, that Google asked if I wanted to review the place I was at, that all my photos were automatically backed up to the cloud, that I got a ‘This day 3 years ago’ notifications every once in a while, that I received personalized notifications about the bills that were unpaid and the events that were due, like magic!
And all these years, I’ve heard about numerous leaks, activists exposing unethical government secrets and mass surveillance and I was always disconnected from it. When Airtel & BSNL were injecting adverts into my web pages, I was okay with it. When Google or Whatsapp changed their privacy policies, I readily accepted the new ones, after all I’m sure they value their users, and decide in their best interests, right? After all, what do I have to hide?
Now, I consider myself a huge fan of free and open source software, and in the open source world, you readily trust the software or content, not because you personally trust the people behind it, but because the code is subject to scrutiny by fellow community members and as a result, the chances of using an open source software that is a malware or a back-doored Trojan is essentially zero (such attempts are readily caught. (meta meta: is this a survivor bias?)). I remember the heavy criticism of Ubuntu for logging the search keywords of it’s users for serving them targeted ads which eventually led to elite members of the open source community advising against using Ubuntu and RMS calling it a ‘spyware’. But what Ubuntu did is only tiny bit as harmful (they did put an option to opt out of this ‘shopping lens’, or uninstall it altogether) as some of the tools and services we use everyday. And that is what I realized in the past month.
From here, it is about how I turned 180 degrees and started to care about privacy and anonymity more than ever, how I became paranoid about the data that I publish online and think twice before registering for an online service, or visiting untrusted websites without a VPN. If you feel this is of no interest to you, I urge you to close this tab after watching the following video. The message is very powerful and I’d like you to give yourself sufficient exposure to the problem before deciding if you want to care. You may continue reading if you would like to learn about my decision and what led me to it.
Let’s start with the most obvious question…
Why now?
The anticlimactic answer is, better late than never. This article isn’t the result of a single blog post that I read or any specific incident. It is a cumulative result of the critical exposure I’ve had in the past month or two, and a subconscious exposure of the past few years. I had this on my mind from some time, but laziness is what I’d call it. Who wants to give away the convenience of synced devices and automatic backups! I’m fortunate enough to have a paranoid friend around who doesn’t use many (any?) social networking sites and online services. All he has is probably a ProtonMail email address, and he’s just as active on the Internet as I am. I always considered his view of privacy a personal preference, a subjective view of the world, not an objective truth about the Internet and companies based on Internet. But recently, the more exposure I’m getting about the way Internet giants collect and use my information, government surveillance etc, the more I’m moving away from using their services. It isn’t about if someone is watching me while I use the Internet, which no one probably is, given my uninteresting Internet activities. It is the possibility that at any given moment someone/something could watch me, without my consent, store tonnes of meta data about me for use 15 years from now, and I might lose the basic right to privacy that I always took for granted, is what makes me uncomfortable.
However I don’t expect anything to change when I make a switch. In most cases, nothing would change for me, as an individual who accesses and relies on the Internet everyday. Free and open source alternatives exist and it is a matter of hours (if not days) to make a complete switch from proprietary to open source software. But now, I’m leaving a lot less footprints in random server logs and by using open source whenever possible, I can narrow down the number of malwares and spywares I carry around with me in my phone or laptop. And something I really need to emphasis on, a spyware is not necessarily installed by just a third party malicious user. OEMs ship spywares all the time (tampering preinstalled TLS certificates and performing MITM attack to show ads, now that’s dark). All this is without even mentioning the humongous quantities of crapware these OEMs ship their products with, widening the attack surface for a third party adversary. All of this can be mitigated if you control what’s installed on your devices and choose what services to use.
If you want to know more about what sort of threats to privacy exist around you, you might want to check out this amazing course by Nathan House titled ‘The Cyber Security Course – Hackers Exposed’. Don’t get intimidated by the title, it is for anyone who wishes to understand the threat landscape so that he can take the necessary steps to ensure adequate security and privacy according to his needs. Nathan does a great job at putting the key points in front for you to decide rather than feeding you his opinions. Highly recommend his course.
Is the threat real?
This question arises in the minds of people when they hear about issues like Privacy and Global Warming (I was surprised to find a good number people think Global Warming isn’t real). Is this real? Or is it one of those hyped-stories that would fade away and everything will get back to normal once media stops covering it. Let me start by confessing that it was in the last month that I read the terms of service of any company I used online for the first time, and boy I was surprised. I agree that reading ToS is boring, but it really is critical to ensure a peace of mind when you use a service. If you’re still not sold, check out this amazing site called tosdr.org or Terms-of-service-didn’t-read which summarizes the ToS of popular services and rates them from class A (good policy) to class E (bad policy) and the key reasons supporting the rating. The data is a bit outdated, but you do get a general sense of the corporation’s privacy structure. And to be honest, you don’t need any of this. All you need to do is keep your eyes and ears open and assess the data you’re about to give to the next application you download from the market. Take Whatapp’s ToS for example, the service which promises that the messages are end to end encrypted with Signal Protocol. Sure, they are. And there’s no doubt in my mind that Whatsapp is one of the most secure messengers we have with us today. But privacy and security are two very different topics to discuss, both equally important (a good read here). And when it comes to privacy, it is not our messages or content that companies usually target. It is our meta-data. Here’s what Snowden tweeted about it.
Are your readers having trouble understanding the term "metadata"? Replace it with "activity records." That's what they are. #clarity
Now there are a lot of articles on thistopic and hence I don’t plan to get into it. To quote a key point from one of the articles about what meta data really is,
They know you rang a phone sex service at 2:24 am and spoke for 18 minutes. But they don’t know what you talked about.
They know you called the suicide prevention hotline from the Golden Gate Bridge. But the topic of the call remains a secret.
They know you spoke with an HIV testing service, then your doctor, then your health insurance company in the same hour. But they don’t know what was discussed.
This example should not imply Whatsapp is the only or worst offender, it is just one of them that I’m familiar with (and use personally). I assume that you can read and decide for yourself.
Why did I care?
“But Government Surveillance is a US problem!”, “My nation doesn’t come under 5, 9 or 14 eyes“. Sure, but don’t we all use the same Internet? I don’t need to emphasis about the importance of Internet in matters of freedom of speech, how it has nullified international borders to connect people of similar interests and how many revolutions it has started. I don’t need to mention that Internet is a second home to political (h)activists and dissidents, a place where they can express themselves to the masses. I certainly don’t need to mention what I personally feel about the Internet which you probably know by now. And I’m not even getting started on how, even if you don’t belong to any of these ‘eye’ nations, most of your traffic is still getting routed through them.
The way I think about it is, a full disclosure sometimes becomes necessary to bring about a change in the system that is broken and is resisting a fix. This was one of the highlights in Keren Elazari’s TED talk on ‘Hackers: Internet’s immune system’
Precautionary measures I took
I adopted some defensive measures to try out this new Internet lifestyle, applying the learnings from the past couple of months, since it wouldn’t make sense to not do it after this exposure. This is quite experimental, so try out what works for you, the way I’m doing. A word of caution though. This list would not cover you from threats and privacy breaches from third party adversaries like cyber criminals, who might choose targets more specifically (like sending a malware via email or infecting your local network). The best (and in many cases, only) defense against it is to keep your systems (laptops, mobile phones) up to date with the latest security patches. Did I make it sound important enough? KEEPING YOUR SYSTEM SOFTWARE UP TO DATE AND PATCHED IS THE BEST THING YOU CAN DO TO STAY SECURE. Sorry for screaming. Okay, now back to the measures I took.
Flashed LineageOS on my Phone – Almost stock Android, plus more control over what I install (note: rooting, flashing, installing from unknown sources etc potentially opens a huge security hole in itself)
No Google Play Services – The suite of Google apps such as Gmail, Youtube, Docs and Drive are optional, and I chose to not install them
Gave up my G Suite subscription. So no synced devices and automatic photo backups. (Remember to ‘Takeout’ data before leaving)
Turned off port forwarding, DMZ, UPnP and any other service on my router that might expose any of my internal devices to the Internet
f-droid for free and open source Android apps, also there are plenty of closed source apk repositories that don’t require a Google account.
DuckDuckGo as the default search engine across all devices
Debian or Arch linux on desktop, as recommended by Nathan House, provides a good mix of active development, security, support and speed, although you can pretty much choose any good distro depending on your taste and harden it.
Signal Messenger on Android/iOS for Whatsapp like security and usability minus the meta data issues
Firefox Focus as the primary browser on phone, except when explicitly wanting to store history, in which case Mozilla Firefox
Mozilla Firefox on desktop, and Chromium as the secondary browser. Google Chrome is a better browser imho, for it supports a lot more content types than Chromium does out of the box. Not to mention better updates and security. Boils down to your personal preference, really
And the rest of the goodies you get with any nice distro. (Must admit that I haven’t found a Google Drive alternative yet)
Lastly (and optionally), encrypted mail providers like ProtonMail for secure email and a good VPN (such as Mullvad, or Tor for that matter, but make sure you read the differences) for use when on public Wifi hotspots
Needless to say, that is what I’m using right now, and kind of recommend. Except for the couple of options on the top, I’m sure most of you are familiar with (and probably use) the rest. If yes, that awesome. That’s a win for the free and open source community. And I’m not affiliated to any of those! Haha
Is any of this necessary?
“You are overdoing it!” as my friend exclaimed. I totally agree, and to be really honest, it is not just about privacy at this point, it is about enjoying the new world that I’ve found, exploring the corners and trying to fit in. I believe that open source shouldn’t feel like a compromise. It should be a pleasant experience for everyone who uses it, whether or not they consciously care about it being free and open. I am sure not everyone is so willing to give away convenience for the sake of some principles and ‘freedom of the web’, and that is totally fine. As long as you take the decision of giving away your data and are okay with it for the rewards it comes with, and not let a corporation decide it for you, I’m no one to tell otherwise. I’m here to tell you that there’s a world out there that represents the open and free nature of the Internet, and it is not at all difficult to convert. I did, and so can anyone.
Links from the post
Aggregating all the blog / additional information post links with their titles from the above text here.
Today we’ll see how you can home brew a cloud (in the local sense) storage solution that would be free to use, quite faster than an Internet based one, secure enough from any outside the network intrusion and customizable.
But why do you need to go to such lengths when you can easily create an account on Google and get 15 gigs of free storage. Well, first of all, data that we generate is increasing significantly each day. We have multiple devices with us, most of them with ~16-64 GB storage, which is not at all good enough. Then while our notebooks are getting faster with solid state drives, they are still costly to use for all of our needs like storing tonnes of movies and music videos, that is, if you are still left with space after cramming up your disk with camera pictures. If you opt for a premium account at Dropbox or Google Drive, it will easily cost you ~$100 a year recurring, the cost which can get you a 2 TB WD external hard disk.
Then there is the speed issue. At least here in India, we are deprived Internet connection faster than 2-4 mbps. Most of the times even less than that. Even if we considered the option of backing to an online cloud storage, the bandwidth prevents us from efficiently using what already exists for free. When using an local cloud, the bandwidth is only throttled by your equipments, and most of the times you can easily get ~40-60 mbps, which is fine.
The last issue, depending on how you see it, is the most and the least important. Security. If the files are going to be random movies and music videos, you might not be much worried about some hacker breaking into your cloud storage provider and downloading them, but on the other hand, if the files contain any kind of sensitive personally identifiable information, then you would worry. But having said that, I would always choose a secure storage solution from insecure ones if given an option, even if the data was not at all sensitive.
Things you’ll need
Now that we’ve discussed some merits and demerits, lets talk about building the thing. The things you’ll need are,
Raspberry Pi (with all it’s setup accessories), with Ethernet port
Hard disk, any capacity, with SATA to USB converter
Wireless router
Ethernet cable or Wifi adapter
USB power hub [in some cases]
Setting up the hardware
Connect the hard disk to the Raspberrypi
Boot it up and login via ssh
Run sudo fdisk -l and make sure the hard disk is shown. Note the device name (/dev/sdb or similar)
If not, try usb power adapter
If it is showing, we’ll have to make sure it mounts to the same location each time we boot up.
Create a folder for the mount point. I’ll be using
/var/www
It would be advisable to use a separate low privileged user for the process, since we will be changing the user home later on.
sudo chmod 775 /var/www
and
sudo chown your_username /var/www
to set the permissions for reading, writing and executing.
sudo blkid
and note the uuid for the external hard disk. Copy it.
Now we need to make the mounting occur each time we boot the pi up. Open the fstab file by
by editing the UUID, mount location and file system and keeping other things constant.
Reboot the pi, and your /var/www should now be pointing to the external hard disk. If so, you are done with this part of the tutorial. If not, check what did you miss. Also make sure you are able to read and write files to that directory from your user account. If not, recheck the steps, Google for solutions or comment for help.
Setting up the FTP server
sudo apt-get install vsftpd
to install the vsFTP server.
Open the vsftp configuration file by
sudo nano /etc/vsftpd.conf
The would be a lot of options. Just go through and make sure the following lines are there and not commented. If not, add them.
After saving (Ctrl + x and then y) and exiting, restart vsftpd by
sudo service vsftpd restart
Lastly, change the user home to the FTP root, so that you’ll directly get into the FTP server’s root on logging into the FTP client.
sudo usermod --home /var/www/ your_username
If all went well, we have a 100% working local cloud storage running off our pi. Now, since not everyone would like to login with terminal each time they wish to access the cloud, I make some customizations to make it easy for even my Mom and Dad to use the cloud.
On the desktop, download and install filezilla.
sudo apt-get install filezilla
should do it on deb derivatives. Create a launcher icon that triggers the command
On the mobile phones (we have droids, three of us), I used the ‘add ftp server’ option in the ES File Explorer and created a shortcut on the home screen with the widgets menu. Hence, accessing the cloud was nothing more troublesome than accessing a local folder on the phone.
Now I have my very own, secure, high speed cloud storage solution for all my devices and also for the family. It is really convenient and building a custom case for the thing, it looks pretty badass.
