Rails Camp East 2016

What Even Is Rails Camp?

Rails Camp is not so much about Rails, or even programming per se. Rails Camp is an opportunity to hang out with other programmers. You can write code if you want. You can also play and be completely offline. I did the latter.

Before

I saw how great the US West Rails Camp looked and had wanted to go. I saw Elle at RubyConf in San Antonio, and she asked if I’d go to the East Coast Rails Camp. I looked into it and registered when I got home. I knew a couple of other people who were going, but not a lot.

Lachlan and I had shared some cocktail recipes and photos through the months. Dark and Stormy is one of my favorites, and he bought a bottle of rum while in Australia to bring back for us to make D&S. I am still touched by that.

As Rail Camp got closer I started seeing some more people talk about it. I was looking forward to seeing some old friends and getting to know people better. I was also looking forward to meeting new people.

We were all invited to a slack channel where we talked about how excited we were and started loosely organizing some things like running, s’mores supplies, and music instruments.

Friday

There was a bus that picked up at JFK airport and Bryant park in Manhattan. I flew into LGA and took a taxi to Bryant Park. Well, I took a taxi almost to Bryant Park. I can’t deal with NYC taxi drivers.

Jessica Suttles claims the cool kids went to JFK, but I think Bryant Park was where the real action was. Nerds with suitcases and backpacks started filtering in and eventually started settling on a bar’s patio.

Rails Camp had begun!

As more people filtered in friendships were made, hugs were given, and conversations picked up where they had left off. The bus arrived and we piled in with the JFK kids.

We got to camp and got our beds. The facility was awesome. I actually slept great all three nights. Dinner was shortly after we got there. The food was all very good, too. After dinner we talked a bit about how to do the Rails Camp thing, and then started playing games and visiting.

Saturday-Sunday

A few of us were runners, and we organized a morning run. After that a group of people had organized a hike. Not all of us got lost on the hike.

There was canoeing, which I regrettably did not do. There was a wood-fired sauna. There was a music room upstairs with instruments. We had the option to sign up for blacksmithing, which was awesome. There were talks throughout the day on whatever you wanted to talk about. At night there were campfires, movies, karaoke, and of course more games.

Blacksmithing!!

Blacksmithing!!

Dark & Stormy supplies

Dark & Stormy supplies

SO MUCH RUNNING!!!

SO MUCH RUNNING!!!

Morning at camp

Morning at camp

Joe with ukulele

Joe with ukulele

It was basically a weekend to disengage with the pressures of shipping code and talk and play with other developers. It was absolutely reinvigorating.

Monday

We had to be out of the cabin by 8, so Monday was a little abbreviated. I still got a good run in and made it back in time to get cleaned up and have breakfast. Back on the bus and back to reality.

It was the most amazing experience. Everyone was incredibly open and kind. I’m super bummed that I can’t make it to the next Rails Camp West.

5 Comments

Filed under programming

My Twitter Bot

Almost a year ago I wrote about putting a Twitter Bot On Heroku. A few months ago my bot stopped responding. This was about the time that Heroku changed their billing plans, but to be honest I don’t know why it went offline.

Today I moved it over to EC2, and I am going to lift the veil and reveal who my Twitter bot is.

EC2 Setup

I picked an Ubuntu image because I prefer Ubuntu over Centos/RHEL. This doesn’t need a lot of horsepower, so the free tier should be just fine. This is also just a silly twitter bot.

So let’s get the foundation in place:

sudo apt-get update
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

And now we need to install ruby. I don’t like installing a ruby version manager on a server. This is a dedicated host, and I control the deployment environment. So I’ll just install ruby and gems at the system level.

cd
wget http://ftp.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz
tar -xzvf ruby-2.2.3.tar.gz
cd ruby-2.2.3/
./configure
make
sudo make install
sudo gem install bundler --no-rdoc --no-ri

Super. Now we can ruby.

The Code

You could do some fancy deploy with Docker, and I thought about doing that. But this is just a silly Twitter bot. So instead I have just a single standalone EC2 instance, and I’ll clone the repo.

Yes, I wrote a Taylor Swift Twitter Bot.

Change into the project directory and run sudo bundle to install the gems. Bundler will complain about running as root, but we don’t care. We are running with scissors on Twitter.

Make It Go

Now we just need to make this thing run, and do that in such a way that if the server reboots that it keeps running. We use an init.d script for that.

I put this simple script in /etc/init.d:

cd home/ubuntu/swift-bot-tay-tay
dotenv rake

I still have the Rakefile from the Heroku setup, and I put a .env file on the box with the credentials. This is a cheesy script, and there are no “environments” so the dotenv gem works just fine.

*Disclaimer: that may not, in fact, be a proper init.d script. I actually kicked the script off from the command line using nohup dotenv rake & and if the server bounces we’ll just see what happens. We are running with scissors, after all.

Celebrate

It’s fun to play with a completely ridiculous project. I’m also amazed how something as simple and silly as a twitter bot elicits as much of a reaction as it does. @SwiftBotTayTay has nearly 900 followers and regularly gets responses to her tweets. Most of them are positive, too.

That makes me smile.

Leave a Comment

Filed under programming, ruby

Language Comparison Showdown

In my day job I work in a lab environment where we constantly experiment and explore. We recently did a project where we had several calls out to different APIs. Some of the calls were dependent, and some could be concurrent. We typically reach for Ruby when we need to stand up a server because it’s a language that we can move quickly in. When you work on several different projects in a quarter, moving fast is something we value and optimize for.

However, Ruby is not good at concurrency — at least not in the way we needed for that project. We reached for Scala in that project, and it worked great. One of our team members has done a fair amount of work in Java-land and Scala, so we had a good bridge and it was pretty fun.

When that project wrapped I was asked to take a look at how that server would look in a couple of alternative languages. That kicked off some conversations that ended up with this apples-to-apples comparison across several languages.

The project for this experiment is a simple one that is the foundation of most of what we do on the back-end in the lab: consume some API and do something with the JSON response.

I created a simple Rack app. Rack is a very thin ruby server. The app serves a JSON payload with a string, integer, floating point number, and date. It’s not a fancy multi-threaded webserver, and it’s running locally. The response time for a single request is 0m0.030s (according to time curl http://localhost:9292).

Here is the server repo: https://github.com/barrettclark/language-comparison-server

I wrote clients in the following languages:
* Clojure
* Elixir
* Go
* Node
* Python
* Rust
* Scala (Finagle)
* Swift 2.0

I think all of these languages handle concurrency to one degree or another. I didn’t really test that aspect of the languages in this exploration. This was more about just getting up and running, and seeing how it felt to just issue the request and handle the response. For the most part these are not languages that I have actually used.

That bears repeating: I don’t really know most of these languages. I’m just some guy who made a simple thing in a lot of languages. I wanted to see how easy it is to get up and running with these. The example isn’t completely contrived, but it’s also not necessarily how you would actually do stuff in the given language. I’m OK with that. I was looking to write code that was as idiomatic as I could muster using the standard lib to the extent possible to compare as apples to apples as possible.

And with that, I give you a big table:

Language Typed Compiled Paradigm (Wikipedia) Ceremony Project Setup 100 Requests
Clojure No Yes Functional High lein new app foo 0m3.049s
Elixir No Yes Functional Low mix new foo 0m0.633s
Go Yes Yes Imperative Low Manually create file(s) 0m1.359s
Node No No Event-driven High npm init is the simplest form 0m23.823s
Python No No Functional, OO, Imperative, Procedural Low Manually create file(s) 0m7.319s
Rust Yes Yes Functional, OO, Imperative Low cargo new foo --bin 0m1.287s
Scala Yes Yes Functional, OO, Imperative High Manually create file(s) 0m3.978s
Swift Yes Yes Functional, OO, Imperative High Create a new Xcode project (for now) 11 ms

Running the benchmarks:

  • Clojure: lein uberjar && time java -jar target/uberjar/client-0.1.0-SNAPSHOT-standalone.jar
  • Elixir: time ../../repeat.sh mix run lib/client.ex
  • Go: go build && time ../repeat.sh client-go
  • Node: time ../repeat.sh node index.js
  • Python: time ../repeat.sh python client.py
  • Rust: time ../../repeat.sh target/debug/client
  • Scala: ./sbt universal:packageZipTarball && tar xzvf target/universal/client-1.0.tgz && ./repeat.sh ./target/universal/client-1.0/bin/client
  • Swift: ./client.swift

A note on Clojure and Scala: I created loops that iterated 100 times inside the compiled code. The JVM startup time is about 2 seconds, so calling into the compiled code cold each time incurs that additional cost. The total time for each was just over 4 minutes when you include the JVM startup in each of the 100 calls.

Similarly, with Elixir, each time to run mix run it fires up the Erlang VM. The total time for 100 full executions was 1m11.299s.

Ceremony refers to the number of hoops you have to line up in order to make code in that environment. A JVM-based language definitely has a lot of hoops to jump through. Dealing with Xcode is a lot of ceremony. Node, with all it’s package and library management, is a lot of ceremony. Rails is also pretty high ceremony now, but I didn’t use Rails in this experiment.

Some Thoughts On All These Languages

Clojure

The repo: https://github.com/barrettclark/language-comparison-client-clojure

I’ve never done LISP. I didn’t use RPN on my HP calculator. I’m not really all that into Java and find that ecosystem a symphony of sadness. In fact, while playing with Clojure I updated openssl, which totally hosed the entire Java ecosystem on my laptop. Leiningen couldn’t pull packages from the Maven repo. I reinstalled Java, and all was right again.

If I worked with someone who was into Clojure I’m sure that would help a lot. Community is super helpful. I don’t, though, and it hasn’t really grabbed me yet.

Elixir

The repo: https://github.com/barrettclark/language-comparison-client-elixir

I first looked at Elixir a couple of years ago when Dave Thomas got excited about it and released his book (as an eBook originally). It was my first taste of functional programming. I did a few exercises on Exercism, but never had a practical application to build. I’ve wanted to come back to it.

On the first pass through this exercise I used HTTPoison to issue requests and Poison to parse the JSON. I found them extremely cumbersome, and felt like maybe Elixir wasn’t for me.

I mentioned that I found Elixir cumbersome on Twitter, and immediately had a handful of replies asking why. They were genuine questions, wondering what it was that didn’t work for me. Remember when I said that community is important? Elixir has a fantastic community who cares deeply about making the language feel good — to the point that confusing error messages are considered bugs.

I took another look with a more simplified solution. It felt good. I also had a handful of people contribute to that repo, which was I really appreciated.

Go

The repo: https://github.com/barrettclark/language-comparison-client-go

At first Go didn’t really strike a chord with me. It felt weird how simple and small the language was. I missed having constructs like map and reduce when working with a collection. Go relies heavily on pointers.

Side effects are a first-class citizen in Go.

Go is also the only language I’ve worked with that cares where you put your code.

However, I think I’ve come around on the language. It is simple, and incredibly powerful. It’s designed to help developers move fast. I want to do more with it. It’s also really easy to parse JSON in Go.

Node

The repo: https://github.com/barrettclark/language-comparison-client-node

I don’t hate Javascript, but I also don’t want to have “one language to rule them all”. I find grunt, gulp, bower, npm, whatevs overly convoluted.

I come from a Ruby background. Convention is a big thing in the Ruby community. I’ve found it really hard to discern a convention in Node.

Python

The repo: https://github.com/barrettclark/language-comparison-client-python

I wanted to go with each language’s standard lib to the extent possible, so I stayed away from Django. I still don’t know how to actually make a “python project”.

The code was pretty easy to write. The standard library looks pretty powerful. The onboarding documentation is a bit of a wall of information (once you find it). It’s a fine language. Being a rubyist, I would probably stick with Ruby when I wanted to use a scripting language, but I believe in horses for courses. There is definitely a place for Python.

Rust

The repo: https://github.com/barrettclark/language-comparison-client-rust

I was blown away with the Rust onboarding documentation. This is another language where you can tell that the community really cares. They also consider confusing error messages a bug in the Rust community. The language is still really young and changing rapidly. It’s been fun to play with.

Scala (Finagle)

The repo: https://github.com/barrettclark/language-comparison-client-scala

Scala, like Clojure, sits on the JVM. It uses a slightly different build process (sbt) that worked better for me than lein. I also work with someone who knows Scala, and that definitely helps take the pain out of lining up the pieces.

From my brief experience with Scala, the base standard library is really just Java. Idiomatic Scala seems to come in with additional layers. We used Finagle at work, and I ended up using that in this experiment.

There is a lot of ceremony and manual setup, but once you get the pieces in place getting and parsing JSON is really nice (especially with Argonaut). A downside of the Scala/Finagle layer cake is that documentation can be a challenge. It was hard for me to discern where to look and how to ask the right questions in The Google.

Swift (2.0)

The repo: https://github.com/barrettclark/language-comparison-client-swift

Later this year we will be able to run Swift on servers. It’s a bit of an odd fit in this comparison. You can run it from the command line, but you wouldn’t. I wasn’t able to get a good time benchmark from the command line, either.

Still, I really like the Swift language. Swift 2.0 has some good improvements. I’ve tried several different approaches to making HTTP calls in Swift. I used NSURLConnection.sendAsynchronousRequest:queue:completionHandler: for a long time. I’ve used Alamofire, which is nice but a really big library. SwiftHTTP is also a nice library that’s a lot simpler. JSON parsing can be done several different ways, too.

A pattern that I really like uses the promises/futures pattern. BrightFutures is a pretty nice library for that. Add in Thoughtbot’s Argo (like their Scala Argonaut) library to parse JSON, and you’ve got a really nice setup.

Xcode still struggles when you go too far into the FP rabbit hole, so there’s that. Still, I’d much rather build a native app than a responsive HTML site, in part because I really like Swift.

Conclusion

It’s fun to look around and see what other communities and languages are doing. I’m still not really comfortable in a FP world, but I like it.

I think my Power Rankings (in order of what I would want to use on my next back-end project) from this experiment are:

  1. Scala Finagle
  2. Go or Elixir
  3. Rust
  4. Node
  5. Python
  6. Clojure

I’m actually torn between Finagle, Go, Elixir, and also Rust. I work with people who do Scala and Elixir, so they have a bit of an advantage. I also really want to play more with Go and Rust. You can deploy both Go and Rust to Heroku, so maybe I’ll sneak them in on a smaller project. Swift doesn’t make the list yet because it doesn’t run on the server — yet.

This Is All Arbitrary

A final thought about all this. I did a very simple version of a very common task that I do at work. The benchmarks aren’t perfect. What do I even know about any of these languages? This was an experiment. I appreciate that I get to do these sorts of things.

The next time a project comes up, we will have some more talking points to help guide technology decisions, and maybe you will find this helpful as well.

Thank you.

8 Comments

Filed under programming

Using Ruby To Make iCal Events, Part 2

Earlier I discussed creating calendar events with Ruby to take the pain out of creating a lot of events. I just finished up a script that created 126 events in seconds. Here are the relevant bits (in addition to the earlier post).

Input

My input CSV looks like this (below). It’s Hal Higdon’s Intermediate 1 plan.

Date,Week,Day,Description
6/8/2015,1,MON,Cross
6/9/2015,1,TUE,3 miles
6/10/2015,1,WED,5 miles
6/11/2015,1,THU,3 miles
6/12/2015,1,FRI,Rest
6/13/2015,1,SAT,5 miles (pace)
6/14/2015,1,SUN,8 miles
6/15/2015,2,MON,Cross
6/16/2015,2,TUE,3 miles

I copied the grid at the bottom and pasted it into Excel. I created another tab to handle the ETL. You can highlight a row and paste special (right click shows you this option) to transpose the cells. That allows you to take a row and paste it as a column. Yes, I can Excel pretty good. Excel sucks at regex though, so I let Ruby handle teasing out the meaning of each of the workouts.

The Code

  def read_input(infile)
    CSV.foreach(infile, :headers => true) do |row|
      m, d, y     = row['Date'].split(/\//).map(&:to_i)
      date        = DateTime.civil(y, m, d, 7, 0, 0)
      description = row['Description']
      next if description == "Rest"
      minutes = if description.match(/^\d{1,2}/)
                  description.match(/^\d{1,2}/)[0].to_i * 10
                else
                  30
                end
      event = Icalendar::Event.new
      event.dtstart  = date
      event.duration = "P#{minutes}M"
      event.summary  = description
      event.comment  = "Event created with Ruby script"
      @cal.add_event(event)
    end
  end

It’s not pretty. That method is doing a lot. The event comment isn’t saved either. That’s OK. I still got what I need — a whole lot of events that I didn’t have to type myself, and a script (and process) that I can use moving forward.

Win-win.

Leave a Comment

Filed under programming, ruby

Using Ruby To Create iCal Events

Calendars can be so tedious, so I hacked mine.

Anytime we have a new season begin, or I start a new running training plan, I have a bunch of events that need to be created. Usually you just sit down and take the half hour (or whatever) to bang all the events out. You can copy and paste events in iCal from day to day, which helps. But it still sucks.

So I finally sat down and figured out how to script it.

Gemfile:

# A sample Gemfile
source "https://rubygems.org"
ruby '2.2.2'

gem "icalendar"

Script (make_events.rb):

require "rubygems"
require "bundler"
require "csv"
require "date"
Bundler.require

### Create iCal ics file with multiple events
class EventCreator
  attr_reader :cal

  def initialize
    @cal = Icalendar::Calendar.new
    make_events
  end

  def make_events
    dt = DateTime.civil(2015, 6, 6, 0, 7, 0)
    3.times do |i|
      event          = Icalendar::Event.new
      event.dtstart  = dt + i
      event.duration = "P30M"
      event.summary  = "Test Event #{i}!"
      @cal.add_event(event)
    end
  end

  def to_ics
    File.open("scripted.ics", "w") { |f| f.write @cal.to_ical }
  end
end

if __FILE__ == $0
  calendar = EventCreator.new
  calendar.to_ics
end

So all you have to do from the command line, after you’ve run bundle install is ruby make_events.rb. It will create a single ics file that you can import into iCal. When you open the file you will be asked what calendar you want to import the events into.

Note: The gem does not appear to support “full day” events. That’s ok. Next up for me is to create a running training plan given a CSV file. I’ll post that next.

4 Comments

Filed under programming, ruby

Roast Pork Loin

I haven’t done a cooking post in a while. I did a roast pork loin this morning for mother’s day that I was pretty happy with. Here are the details:

Roast Pork Loin
Easy, tasty, and juicy pork loin
Write a review
Print
The rub
  1. 3 Tb brown sugar
  2. 2 ts garlic powder
  3. 2 ts chili powder
  4. 1/2 ts black pepper
  5. 1/2 ts dried oregano
  6. 1/2 ts salt
  7. 1/2 ts annatto seed (optional)
  8. A Pork Loin (or tenderloin or whatever).
Instructions
  1. The pork in this post is a large loin, about 6 lbs. I doubled the above recipe.
Prep
  1. I took peppercorns, sea salt, and the annatto seeds and ground them all up together. The salt will disintegrate quickly, so maybe add it towards the end of the grind. Or just use kosher salt.
  2. The annatto seeds are totally optional. I've started using them whenever I use chili powder. Both are red.
Rub
  1. This is a spice rub, and everything will be just fine if you do whatever you want with it.
  2. Put the rub all over the pork, wrap it in plastic wrap (air tight), and let it sit for a while. I let mine sit overnight. Longer supposedly yields more flavor. Do what works for you. It'll be fine.
Cook Time
  1. I cooked mine on my Big Green Egg (large). I put a single (one) dry piece of pecan in the coals once I got them set at around 350. Plate setter for indirect roasting. Meat in with a thermometer probe.
  2. Cook until the temp hits around 140. I think I took mine off at 145. I've seen estimates of a half hour per pound, but mine cooked within ~2 hours for 6 lbs. Take the pork off anywhere from 135 to 145 and you should be fine.
Rest
  1. Keep the thermometer probe and wrap it in foil and/or plastic wrap. Wrap that in towels. Put that in a cooler and put more towels or old sheets on it.
  2. The temperature will climb while the meat rests. Mine got all the way up to 160 and was still above 140 a few hours later in the cooler.
Notes
  1. I put a disposable pie pan with 2 or 3 cups of water in it on the plate setter to get a little more moisture in the egg.
  2. Enjoy.
Cooking. Code. http://cookingco.de/

The recipe was adapted from:

http://www.food.com/recipe/grilled-pork-tenderloin-with-texas-rub-149784
http://eggheadforum.com/discussion/1114265/smoking-pork-loin

1 Comment

Filed under Big Green Egg, cooking

RailsConf 2015

I recently had the privilege of being a speaker and a guide at RailsConf in Atlanta.

First, a little about the Guide program. Guides are mentors to Opportunity Scholarship recipients, who are generally new to the Ruby/Rails communities or maybe new to programming in general. The guides are there to help make RailsConf a little easier and less intimidating. There are a lot of people, and I confess that I still get intimidated at times.

We were paired before the conference, and I reached out to my scholar (Eduardo) in advance to get to know him a little. There was also a Guide Mixer the night before the conference began that gave everyone an opportunity to meet each other. It’s always fun to be around a group of people who are really open and want to help. That’s one of the things that I love about the ruby community in general.

I will also confess that I’m generally fairly introverted, and being a guide was also a good way to help me break out of that shell. There were several times when I would see someone and introduce them to Eduardo. It was also great just meeting the fellow guides and having them as a peer set.

The guide program was great. Be on the look out for it at RubyConf.

OK. Now let’s shift our focus to the conference itself.

There were somewhere between 1500 and 2000 people in attendance. I wanted to get out of my introvert/ambivert shell and get the most out of being around so many people in my community. I got to Atlanta on Sunday to see some family and was at the hotel by mid-afternoon. I had already made a connection with someone via Twitter about going for a run during the conference, and we decided to meet in the lobby Sunday afternoon to talk shop and see about some dinner.

This is where the magic really began. At any given point during my time there I was with someone who knew someone who knew someone…. Ad hoc groups would form and start talking and before you knew it you’ve made new friends and are going somewhere to eat, or go to a talk together, or make plans to catch up later that evening. You couldn’t turn around in (or around) the hotel without seeing someone (or a group of people) from the conference.

I was a little nervous about my talk. I gave a fairly technical talk that involved a lot of code. Would there be some hotshot in the crowd who wanted to show me up? No. It went well (I think). I had some great questions and conversations after.

The last evening there was a giant mass of people who all wanted to be together, so we split up into 3 large groups. The restaurant wasn’t able to get any of us close, sadly. It was still a lovely evening. I hated saying goodnight to everyone, but I was spent.

There were some great talks throughout the week. 6 tracks going at the same time with 1 being hands-on workshops. I’m always inspired seeing what people are talking about and exploring. It was even more fun seeing my friends giving their talks. More than any of that, though, was an overwhelming sense of community. I made a lot of new friends and deepened existing friendships.

I look forward to seeing my ruby friends again – sooner than later I hope.

1 Comment

Filed under programming, ruby

Team Vim

I learned vi a long time ago, but I never could make it work for me as my primary text editor. About once a year I would try, but it always just seemed like a mess of a .vimrc config file and a pile of esoteric plugins. I also think the idea of remapping keys is silly.

At this point people go usually bananas.

I’ve been fortunate to have the opportunity to go to a few conferences recently. While at Ruby On Ales I recounted a recent conversation on Twitter where there was some, um, fervor expressed. I was talking with Nickolas Means (@nmeans), who said that the key to a vim workflow was really to get a handle on tmux. Then slowly build into vim.

I filed that away and had some time recently to go back to it.

I quickly realized that I liked having different workspaces that I could jump in and out of without having a whole lot of terminal windows open and tabs. That’s actually pretty damn amazing. Being able to split the window into panes and have multiple windows and all that. You tmux nerds are already all over this, of course.

There was still something holding me back from full-on project editing adoption, though. Moving around a big project is a real pain in the ass in vim. I’m not down with NERDTree either. I want a minimalist vim config, and you can see a directory and pick a file in vim already without any additional goofy plugins.

I got the tip that I needed in a workshop session at RailsConf that Eileen Uchitelle (@eileencodes) gave. CTags.

Money.

Now I can jump around in a sane way to related things.

Other things that are pretty killer: I’ve always liked the search and replace functionality in vim, and the ability to jump to lines. Yes other editors also have this, but it makes more sense in vim. There are some other things like not having to have your cursor on a line to edit it or align things.

Anyways, I’m sort of on team vim now. I’m not foaming at the mouth about it or anything, but I think I’ve made it past the “giant pile of esoteric plugins” to “useful tool” stage.

Thanks to Nick and Eileen for your tips. And thanks also to Geoff Petrie (@geopet) for being really cool and offering to do a screen share pairing session to understand my workflow so that we could both learn something from each other. I’ll take you up on that offer at some point, Geoff.

Leave a Comment

Filed under programming

Twitter Bot On Heroku

I made a twitter bot a week or two ago, and it has been a lot of fun watching people’s reactions to it. I won’t tell you what my triggers are or who the bot is because I think that’s part of the game. The bot lives entirely on it’s own. What I will tell you is how to host a Twitter bot on Heroku.

The Bot Code

You’ll use the Streaming API to search for stuff, and then the REST API to post “responses”. I followed this example.

require 'rubygems'
require 'bundler/setup'
Bundler.require

class Bot
  def self.go
    while true
      begin
        config = {
          :consumer_key        => ENV['TWITTER_CONSUMER_KEY'],
          :consumer_secret     => ENV['TWITTER_CONSUMER_SECRET'],
          :access_token        => ENV['TWITTER_ACCESS_TOKEN'],
          :access_token_secret => ENV['TWITTER_ACCESS_TOKEN_SECRET']
        }
        rClient = Twitter::REST::Client.new(config)
        sClient = Twitter::Streaming::Client.new(config)

        # topics to watch
        topics = ['hello', 'world', '#helloworld']
        sClient.filter(:track => topics.join(',')) do |tweet|
          next if tweet.text.match(/#{rClient.user.screen_name}/i)
          if tweet.is_a?(Twitter::Tweet) && !tweet.text.match(/^RT/) && tweet.lang == 'en'
            p "@#{tweet.user.screen_name}: #{tweet.text} (ID: #{tweet.id}) (#{tweet.lang})"
            rClient.update("@#{tweet.user.screen_name} Pithy response", in_reply_to_status_id: tweet.id)
          end
        end
      rescue Exception => e
        puts 'error occurred, waiting for 5 seconds' + e.class.to_s
        sleep 5
      end
    end
  end
end

I put the bot in a class, so it’s a classy bot. It also is a bot that will run FOREVER. I am also using a Gemfile and I set some environment variables for all the secret stuff. You can set those from the command line:

heroku config:set TWITTER_CONSUMER_KEY=SOSEKRET
etc…

Gemfile

The Gemfile is really simple. You just need the Twitter gem.

source "https://rubygems.org"
ruby '2.1.4'

gem "twitter"

bundle init is a quick and easy way to get a basic Gemfile that you can start adding stuff to. Those first 3 lines at the top of the script is how you require the contents of the Gemfile in a plain old ruby script. There is a big world that exists off-rails, and I recommend you check it out.

Make It Go

You now have everything that you need to make it go. You can do it in irb to see how it performs. I would recommend you comment out the line that makes the response until you’re sure it does what you want.

To run it in irb, set your environment variables, or copy the secret stuff into your script, and then run:

irb -r ./bot.rb

That’ll put you at the command prompt in irb where you can run Bot.go and watch it go.

What About Heroku?

Right, Heroku. If you push this as it is to Heroku, it won’t do anything. You need a Procfile. It looks like this:

bundle exec rake

Yes really.

Rake is awesome, and custom rake tasks are the bee’s knees. So let’s make one.

require './bot.rb'

desc "Bot!"
task :bot do
  Bot.go
end

task :default => 'bot'

There is a rake task bot, and the default rake task is set to that, so if you simply run bundle exec rake you get a running bot.

Push those 4 files to Heroku, and you’ve got yourself a twitter bot!

Leave a Comment

Filed under programming, ruby

SQL Is Your Friend, Part 4

So far we’ve talked about the JSON datatype with the first part. Postgres 9.4 brings some new features that enhance it’s JSON capabilities even more, but I haven’t had a chance to play with the new shiny yet.

In part 2 we talked about using window functions. We also made our own functions in both to help extend functionality or DRY up repetitive queries.

I went over a couple of subquery techniques in part 3. These are exciting to me because it gives me even more capabilities to make sense of data at the data level. With windowing and subquerying I can give context to the data in-place, without having to pull it down and write piles of app server code. I like that.

Today we are going to talk about a performance optimization. What do you do when you’ve got the data indexed well, but the query still doesn’t respond in a quick enough time? As a measuring stick, I give a critical eye to anything that takes more than 500 milliseconds. Once it nears a full second I start looking for options.


Materialized Views


Postgres gives you the ability to save (materialize) the results from a query. It’s not a temp table, or even a table, but you can give the data indexes and query against it the same as a table.

-- Store off the query results
CREATE MATERIALIZED VIEW readings_mv AS
SELECT *, LEAD(minor) OVER (ORDER BY reading_timestamp) AS next_minor
FROM readings
WHERE reading_timestamp >= NOW() - '1 day'::INTERVAL;

-- Add some indexes
CREATE INDEX index_readings_mv_on_major ON readings_mv USING btree(major ASC NULLS LAST);
CREATE INDEX index_readings_mv_on_phone ON readings_mv USING btree(phone COLLATE "default" ASC NULLS LAST);
CREATE INDEX index_readings_mv_on_reading_timestamp ON readings_mv USING btree(reading_timestamp ASC NULLS LAST);

Note the name: readings_mv. It’s a materialized view that consists of readings table data. That’s a convention that I picked up along the way to help communicate what database objects are being used. This is helping out your team and also future you.

In this particular example the query needs to run against the past 24 hours of data and do some calculations (windowing). Now we’ve taken the heavy lifting and calculating and done that in advance out of band from the request/response cycle. We can also do more efficient subqueries that rely upon this same data. You don’t have to index the materialized view, but I tend to mirror table indexes (or add new ones where it makes sense).

Check out the SQL Fiddle. Scroll all the way down in the left pane to see the creation of the materialized view.


Refresh The Data


That’s great, but I said that I wanted to query against the past 24 hours and time is a moving target.

No worries, you can refresh the data with a simple query:

REFRESH MATERIALIZED VIEW readings_mv;

Easy peasy. It blocks, though, so while the materialized view is being updated you have to wait. Or you had to wait. Postgres 9.4, which just dropped this week, added the ability to refresh a view CONCURRENTLY.

REFRESH MATERIALIZED VIEW CONCURRENTLY readings_mv;

Here is the Postgres 9.4 changelog (of sorts) and a little more explanation on the REFRESH CONCURRENTLY functionality as it was being included in trunk.

You can put this into a rake task as well:

namespace :db do
  namespace :heroku do
    desc "Load the PROD database from Heroku to the local dev database"
    task :load => :environment do
      if Rails.env == 'development'
        Bundler.with_clean_env do
          config = Rails.configuration.database_configuration[Rails.env]
          system <<-CMD
            heroku pgbackups:capture --expire
            curl -o latest.dump `heroku pgbackups:url`
            pg_restore --verbose --clean --no-acl --no-owner -h localhost \
              -U #{config["username"]} -d #{config["database"]} latest.dump
            rm -rf latest.dump
          CMD
        end
      end
    end

    # https://devcenter.heroku.com/articles/scheduler
    desc 'Update the materialized view(s)'
    task :update_materialized_view => :environment do
      sql = 'REFRESH MATERIALIZED VIEW beacon_readings_lead_lag_mv;'
      ActiveRecord::Base.connection.execute(sql)
    end
  end
end

If you use Heroku you can this up with scheduler to run every 10 minutes. I included an additional task that I find useful to populate my local database with live data.


kthxbai


These are techniques that I find myself using when an app goes from simple CRUD to having to make sense of complex data. I hope that you found this series interesting and useful. I also give you permission to occasionally use raw SQL to take advantage of the features that your database brings to bear. You chose it for a reason, so use it.

Now go make big, interesting things.

1 Comment

Filed under programming, ruby, SQL