If you create a binary raster (1/TRUE for water surface and 0/FALSE for land surface) e.g from NDWI using a threshold (0.3 or 1), than shorelines can be selected with
1) drawing polygons from water (raster::rasterToPolygons(, dissolve = TRUE) in R), and then rasterizing it (raster::rasterize()); or
If you need shareline length, you might use sf::st_as_sf() on the result of rasterToPolygons(), than cast it to lines (st_cast() or st_boundary()), and calculate the length (st_length()).
If you have a gradual, gently sloping coast, you could have a look at the PhD thesis of R.S. Dewi:
Dewi, R. S. (2018). Multitemporal image analysis for monitoring fuzzy shorelines. (331 ed.). Enschede: University of Twente, Faculty of Geo-Information Science and Earth Observation (ITC). https://doi.org/10.3990/1.9789036546331