How to break a server part 2

Well, it happened again. I broke my server.

On Friday afternoon, I was working on someone’s site that is hosted on my server. I had PuTTY and WinSCP opened while I was working.

I remember months ago typing in “whereami” and found out that it’s a real package. At first instinct, I thought it was a command to state what directory you are in. So I decided to install it.

That was the mistake, installing whereami.

The install stopped at about 78% on my screen, and froze. All of sudden WinSCP disconnected but my SSH session was still up at the frozen install. At first, I thought that me indexing a folder killed it.

Luckily, my server is hosted via DigitalOcean. They have a nice little console window that works no matter what. Thank God.

After many hours that night figuring out what the fuck I did, I went to bed in defeat. I worked all that weekend so I had no time to investigate. I decided to create a support ticket to see what they can do since I had no idea.

They told me hours later to run some commands to make sure my network config is correct, and it was from what I could tell. The main issue at that point were a couple of things:

  • All outbound traffic times out while trying to connect
  • Anything trying to connect outside the server fails
  • Since nothing could connect, nameservers died

So I investigated more on Sunday night and found a major issue with my config. The default gateway was set to 0.0.0.0! Ha! I found the issue!

I quickly set the gateway back to what it should be, and heaven’s light shined on me. All was working!

So RIP to the 5 websites and 1 Twitch bot that are all hosted on this server.

Turns out whereami is some networking setup package? I still don’t really know but it reset my entire network config.

Fuck whereami.

~ hyper

How to break a server

If you saw on my Twitter, I may have broken my website on Friday night. I lost file and SSH access to it. How did it happen?

This all started with me moving my main domain over to hyper.lol. I thought I could just change all configs to the new domain and it would work. Some things were broken obviously. So I thought the issue was with file and folder ownership. So I change the directory to the site and did this command:

sudo chown -R www-data:www-data /

To anyone that is stupid enough, don’t run that command above. It WILL break your computer/VPS. That command destroys the file/folder perms making your install break. My thought process was that the / was relative to where I currently was. That was not the case. At that point, I was unable to login at all. The server couldn’t accept my key.

I realized what I did after I typed the command. Luckily, I was still logged in via WinSCP. To fix it, I just went to my root directory and mass changed the ownership back to root. I had to leave it on overnight; however, that still didn’t work.

I ended up contacting my hosting’s support. After about 10 emails, we decided that these were the best steps to fix this:

  • Support would mount a recovery tool. From there, I could access the files.
  • Download everything I want.
  • Support would unmount the recovery tool.
  • I rebuild the server’s OS.

That is what we did, and now everything is working once again.

I also did change my domain over to hyper.lol, since I had the chance to rebuild everything.

TLDR, idiot tries to change file and folder perms but messes up the location on the command. Contacts support to fix.

~hyper