ttyz, the regulator needs a minimum amount of current coming from the output pin to the adjust pin through that R1 resistor. If you increase the resistor more than - let's say probably 150-180 ohm - the current may be too low in some cases and the regulator may not regulate properly. People have had problems with a 240 ohm resistor, which was fine for LM117 but not good enough for LM317 in all situations.
It's recommended to leave that resistor to 100-120 ohm.
Your voltage will be 24v AC rms x 1.414 = 34v +/- about 5-10% (more at low loads/ less at high loads). The bridge rectifier will waste about 1.5-2v so you are left with about 32v.
But that's with the rectifier producing a perfect DC, which is not the case. The capacitors after the bridge rectifier smooth out the output and try to get the DC voltage as ripple free as possible.
The formula to calculate how much capacitance you need there is something like C = 0.75x current / (2 x f x Vripple) where Vripple is how much you're willing to let the input voltage vary, f is the ac frequency, current is the maximum current.
So assuming after the bridge rectifier you have a peak voltage of 32v, if you want the voltage to not go down more than 1v at 1A, then you have C = 0.75 x 1A / 2x50Hz x 1v = 0.75 / 100 = 0.0075F = 7500uF
So at 1A, with 32v peak from bridge rectifier, with about 3300uF capacitance you'll have 30-32v DC. It's not a great idea to put A LOT of capacitance, you'd be "stressing" the transformer. It's common to just accept a small voltage drop of 1-2 volts and use at most about 4700uF for 1A.
Now, LM317 needs about 1.5-2v ABOVE the output voltage to regulate properly, so the best you're looking at when it comes to output voltage is 1.25v - 28v, maybe a bit more if you add a lot of capacitance.
So I suggest aiming for a maximum of 28v on the potentiometer....
The output voltage is calculated with formula Vout = Vref ( 1 + R2/R1) = 1.25(1 + ? / 100) so for 28v = 1.25 (1+ ? / 100) => ? = (28/1.25 - 1 ) * 100 = 2140 ohm.
So now you can do a trick ... you can put a resistor in paralel with the potentiometer to reduce the maximum resistance.
1/Rtotal = 1/R1 + 1 / R2 =>Rtotal = r1xr2 / (r1+r2)
So for example, you can put a 3900 ohm resistor in paralel, so the maximum will be 3900x5000 / 8900 = 2191 ohm.
Last... keep in mind that the lm317 can dissipate about 15 watts with a good heatsink. The power dissipated is calculated with formula P = (Vin - Vout) x Current.
With 30v in (after rectification, capacitors etc), if you want 3.3v out ... P = (30-3.3) x current, so the maximum current you'd be able to get out would be 15/ 26.7 = 0.56A. For 0.8-1A out, you're looking at a maximum of about 15v difference between input and output, and that's with a BIG heatsink (or a smaller one but with fan blowing on it).
A possible solution would be if the transformer has two secondary windings ( 0 - 12 - 24 or 0 - 12 0 - 12) so that if you want your power supply to output 1.25-12v, you switch to the first winding, and if you want more you join the two windings so that you have higher input voltage.