{ Josh Rendek }

If your developing against kubernetes services or want to run your changes without pushing to a remote registry and want to run inside kubernetes:

First create a registry running in minikube:

1kubectl create -f https://gist.githubusercontent.com/joshrendek/e2ec8bac06706ec139c78249472fe34b/raw/6efc11eb8c2dce167ba0a5e557833cc4ff38fa7c/kube-registry.yaml

Forward your localhost:5000 to 5000 on minikube:

1kubectl port-forward --namespace kube-system $(kubectl get po -n kube-system | grep kube-registry-v0 | awk '{print $1;}') 5000:5000

Use minikube’s docker daemon and then push to localhost:5000

1eval $(minikube docker-env)
2docker push localhost:5000/test-image:latest

And then you can develop your helm charts and deploy quicker using localhost. No need to configure default service account creds or getting temporary creds.

If you’ve been following kubernetes, you’ll understand theres a myriad of options available… I’ll cover a few of them briefly and why I didn’t choose them. Don’t know what Kubernetes is? Minikube is the best way to get going locally.

The goal of this project was to start with a base directory (in this case The Hidden Wiki) and start spidering out to discover all reachable Tor servers. Some restrictions were placed on this after a few trial runs:

• Only HTML/JSON was parsed/spidered for more links to follow (no jpegs/xml, etc)
• There were a few skipped websites, noteably: Facebook, Reddit, and a few Blockchain websites due to the amount of spidering/time that would be required
• Limited to 10k visits per host so we wouldn’t infinitely keep spidering / some reasonable time frame to finish
• Non 200 OK status responses were skipped

Table of Contents

• Stack & Tools
• Crawl Stats
• Security Headers
• Software
  • Source Code Hosting
  • Build Servers
  • Popular Servers
• Summary

Performance before and after Optimizations

When working with billions of documents in your Elasticsearch cluster, there are a few important things to keep in mind:

• Look at what the big players do (Elasticsearch/Kibana) for organization and planning
• Experiment with index sizes that make sense for your business, don’t just assume 1 index for a billion documents is a good idea (even if you N shards)
• Understand which metrics to monitor when you are performance testing your cluster
• Monitor all points of ingestion: Elasticsearch, Load balancers (ELB, HAProxy, etc), and your application code that is inserting

What do the big players do?

Split by date ranges. Based on your data, decide whether daily, weekly, or even monthly splits are best for your dataset. Elasticsearch reccomends not going over 30-32G per shard based on current JVM memory reccomendations . The reason they reccomend to stay below 32G of ram per shard is that after that, the JVM will use uncompressed pointers which means internal pointers go from 4 bytes to 8 bytes, which (depending on your memory size) can lead to decreased heap available and also increased GC times from the JVM.

If you’ve done any concurrency work in Go you’ve used WaitGroups. They’re awesome!

Now lets say you have a bunch of workers that do some stuff, but at some point they all need to hit a single API that your rate limited against.

You could move to just using a single process and limiting it that way, but that doesn’t scale out very well.

While there are quite a few distributed lock libraries in Go, I didn’t find any that worked similarly to WaitGroups, so I set out to write one.

You’ve raised your file descriptor limits, updated security limits, tweaked your network settings and done everything else in preperation to launch your shiny new dockerized application.

Then you have performance issues and you can’t understand why, it looks to be network related. Alright! Let’s see what’s going on:

1ping google.com
2unknown host google.com

Maybe its DNS related…. Let’s try again:

1ping 8.8.8.8
2ping: sendmsg: Operation not permitted

That’s odd, maybe it’s a networking issue outside of our servers. Lets try pinging another host on the subnet:

I’ve been very happy using InfluxDB with Grafana + StatsD but always wanted a nice way to alert on some of the data being fed into statsd/grafana so I wrote a little tool in Go to accomplish that:

Github: https://github.com/joshrendek/influx-alert

I hope someone finds this useful! It’s got a few simple functions/comparisons done already and support for HipChat and Slack notifications.

Documentation

Influx Alert

This is a tool to alert on data that is fed into InfluxDB (for example, via statsd) so you can get alerted on it.

I was setting up the ELK stack and had quite a fun time trying to get upstart to log to syslog WITH a log tag ( aka: my-application ) so it could be filtered inside Kibana.

Here is a working example for STDOUT and STDERR:

 1respawn
 2respawn limit 15 5
 3
 4start on runlevel [2345]
 5stop on runlevel [06]
 6
 7setuid app-user
 8setgid app-user
 9
10script
11  # Redirect stdout to syslog
12  mkfifo /tmp/app-stdout-fifo
13  ( logger -p user.info -t your-app-tag </tmp/app-stdout-fifo & )
14  exec 1>/tmp/app-stdout-fifo
15  rm /tmp/app-stdout-fifo
16
17  # Redirect stderr to syslog
18  mkfifo /tmp/app-stderr-fifo
19  ( logger -p user.err  -t your-app-tag </tmp/app-stderr-fifo & )
20  exec 2>/tmp/app-stderr-fifo
21  rm /tmp/app-stderr-fifo
22
23  exec ./your-app-binary
24end script

Hope this helps someone else, there as a lot of mis-leading and broken examples on Google & StackOverflow.

Below is some advice and notes that I wish I had when writing Go to deal with high amounts of requests (20k+/second). Have any extra tips? Leave them in the comments!

Kernel Tuning

Step 1 is making sure your host OS isn’t going to keel over when you start making thousands of requests/second or hammering the CPU.

Update /etc/sysctl.conf to have these lines:

1net.ipv4.tcp_tw_reuse = 1
2net.ipv4.tcp_tw_recycle = 1
3net.ipv4.ip_local_port_range = 50000

ip_local_port_range - at the default of 30,000 and not modifying the tw_reuse and tw_recycle properties, we’re effectively limited to 500 connections/second to a server. If this is still not enough you can configure additional IP’s on the server and cycle between them.

Let’s make a function to generate an HTTP client for us using a custom dialer:

 1var DefaultDialer = &net.Dialer{}
 2
 3func GetHttpClient() http.Client {
 4	tr := &http.Transport{
 5		Dial:                DefaultDialer.Dial,
 6	}
 7
 8	client := http.Client{Transport: tr}
 9	return client
10}

Can you spot the bug?

By omitting the Timeout, KeepAlive timeouts in the first example, we’ve introduced a very subtle bug.