Could an agricultural workforce be made up almost entirely of day laborers?
Yes. As an analog, look at the US agricultural system. While not "almost entirely", it makes heavy use of migrant labor, people moving from region to region following various harvests and plantings which need a lot of labor for a brief time.
Source: USDA Economic Research Service
This is an average for US agriculture as a whole, and only for hired workers (vs the owners and full-time employees). Certain very labor intensive crops will require more hired workers than the average. And a fantasy setting won't have the mechanization that the modern world does and thus will need more labor.
Good enough to show it's feasible.
What would the impact be on permanent residents of agricultural communities (both landowners and other workers)?
It's great for the landowners! They get a lot of cheap labor, right when they need it at harvest or planting time, and then once the work is done they can be paid off and disposed of. They don't have to care for those laborers for the rest of the season.
The local (stationary) agricultural workers will be harmed. The influx of a supply of migrant labor will drive wages down. Not only during harvest and planting, but it will have a knock-on effect. The landowners will say "why hire you at $X/hour when I can get four adventurers for that?" So long as there are always a few adventurers hanging around town, short on coin, the threat of hiring them is enough to drive wages down. It doesn't matter whether there's actually enough labor to make good on the threat.
Cheap labor also devalues skilled labor and mechanization. Why train and hire and expensive skilled worker when you can get four unskilled adventurers? They each do half the work, but you can hire four times as many for the same money! Why invest in machinery to do the work when you can hire cheap adventurers?
This would also be used to break up any unions... I guess in a fantasy setting guilds. Don't want to deal with the guild rules or prices? Hire a bunch of adventurers. If guild workers can't get work, and if the guild doesn't have leverage over the employers to make them use guild workers, guild workers will defect and start working for lower pay and worse conditions.
Then there's the issue of having all these migrant workers with little connection to the local community. Once the work is done and they're paid off they're suddenly flush with cash and out of work. This is both a boon for the locals, since they can sell them things, and a bane, since a bunch of bored, young adventurers from out of town might get drunk and rowdy.
Would food be less or more expensive with this type of labor readily available?
Cheap labor drives down production cost. How much so depends on how labor intensive the particular food is.
In an environment with many farms competing with each other, this would drive down the price of food. In a non-competitive environment with a few farms colluding they'd keep prices high and pocket the extra money as profit. The latter is very likely in a medieval setting with no government oversight and no global food transportation. Nobody is going to break up a monopoly and there's no cheap fruit from Argentina to compete.
But at the same time, there would be less money in the local community to buy the food. Instead of local farms paying local workers and keeping that money in the community, the local farms pay migrant workers and their pay leaves the community. So the locals have less money.
