indexes across joins not used for count

I'm running a query to do a count with two joins in it. I've added
indexes to the tables for the join columns, but the explain of the query
doesn't seem to be using the indexes:

Table 1:
   invites (id:int)

Table 2:
   sms_requests (id:int, invoker_id:int, invoker_type:string,
sms_message_id:int)
   Indexes:
     "sms_requests_pkey" PRIMARY KEY, btree (id)
     "index_sms_requests_on_invoker_id_and_invoker_type" btree
(invoker_id, invoker_type)
     "index_sms_requests_on_sms_message_id" btree (sms_message_id)

Table 3:
   sms_messages (id:int, sent_at:timestamp)
   Indexes:
     "sms_messages_pkey" PRIMARY KEY, btree (id)
     "index_sms_messages_on_sent_at_partial" btree (sent_at) WHERE
sent_at IS NULL
     "index_sms_messages_on_sent_at" btree (sent_at)

Query:

SELECT COUNT(*) FROM "invites" INNER JOIN "sms_requests" ON
"sms_requests"."invoker_id" = "invites"."id" AND
"sms_requests"."invoker_type" = 'Invite' INNER JOIN "sms_messages" ON
"sms_messages"."id" = "sms_requests"."sms_message_id" WHERE
"sms_messages"."sent_at" IS NOT NULL

Explain:

  Aggregate  (cost=165914.42..165914.43 rows=1 width=0)
    ->  Hash Join  (cost=92326.82..163534.87 rows=951821 width=0)
          Hash Cond: (sms_requests.sms_message_id = sms_messages.id)
          ->  Hash Join  (cost=32692.53..83674.38 rows=951821 width=4)
                Hash Cond: (invites.id = sms_requests.invoker_id)
                ->  Seq Scan on invites  (cost=0.00..27525.48
rows=1238948 width=4)
                ->  Hash  (cost=20794.76..20794.76 rows=951821 width=8)
                      ->  Seq Scan on sms_requests  (cost=0.00..20794.76
rows=951821 width=8)
                            Filter: ((invoker_type)::text = 'Invite'::text)
          ->  Hash  (cost=48180.24..48180.24 rows=916324 width=4)
                ->  Seq Scan on sms_messages  (cost=0.00..48180.24
rows=916324 width=4)
                      Filter: (sent_at IS NOT NULL)

This is pretty slow, ~5000ms on my development machine. I would have
expected it to be able to make use of the indexes I've created. Any
ideas on what I can do to make this perform better?

Jeremy