Remove Modifiers on Table

I created a modifier for auto incrementing my primary key as follows:

records=# \d users
                                Table "public.users"
 Column |         Type          |                     Modifiers
--------+-----------------------+----------------------------------------------------
 id     | integer               | not null default
nextval('users_seq_id'::regclass)
 fname  | character varying(40) | not null
 lname  | character varying(40) | not null
 email  | character varying(40) | not null
 office | character varying(5)  | not null
 dob    | date                  | not null
 title  | character varying(40) | not null
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)
    "users_email_key" UNIQUE, btree (email)

I recently tried to remove the modifier and it failed because it was
associated with the 'id' column so my question is how do I remove /
delete the modifier so I can delete the sequence I created to auto
increment my 'id' value? I don't want to drop the id column / loss my
column data, I just want to remove the associated modifier so I can
drop the sequence.

Thanks for any assistance.