I successfully made a few time lapse sequences, see the Pacific Bulb Society You Tube Channel.
All these were with a Canon compact Powershot camera. At first I installed CHDK this would allow around 4 hours on a set of 2 AA batteries taking one photo every 50 seconds. Then I added a mains power supply. I never got beyond a couple of days - the camera would eventually crash. Eventually I used an Arduino to trigger the camera - again using the CHDK facility for shutter release via the USB port.
Another variation was using a stepper motor to move the object being photographed - see Stepper. Yet another version uses a Raspberry Pi to control three stepper motors moving a DSLR (pan, tilt, rail). Control is from a browser connected via WiFi to the Pi see Stepper Pi Node.
The problems with this are that the kit is a bit too expensive to lose, it is fragile and requires a lot of power. I was reluctant to turn the camera on and off repeatedly because there is a mechanism for popping up the lens.
So the idea is to find a camera which can be controlled from an Arduino, can be turned on and off and is small and cheap. The whole setup to be tiny and run off batteries for a long time. This is the 'budget', power, size and price.
Later this appeared Mini Spy Camera With Trigger for Photo or Video - Adafruit.
To start with I investigated some very low cost cameras off eBay.
Digital Camera 5MP HD Smallest Mini DV Video Recorder
2.4in 1280*720 Video Car DVR Vehicle Camera Blackbox Night Vision G-sensor F7
From my modest experience it became apparent the eBay market for such cameras consists of vendors who distort specifications and supply fake products. This diminished my enthusiasm. My third eBay purchase not only didn't arrive, but the vendor vanished. Presumably these things are a mass market product and it is worth acting fraudulently.
Other people have done extensive reviews of similar cameras. For example:
Techmoan
Chuck Lohr 808 car keys cameras
I abandoned the idea of getting a cheap camera - I moved on to building one around a Raspberry Pi Zero - see Pi Timelapse Camera.