Estrategia de asignación de precios minoristas de electricidad basada en aprendizaje por refuerzo para sistemas de distribución con múltiples microrredes

Juan Manuel Rey; Camilo Esteban Carrillo Valera; Jhon Héctor Sandoval Manrique; Andrés Felipe Luna Chacón; Iván David Serna Suárez

doi:10.14482/inde.44.01.103.911

Authors

Juan Manuel Rey Universidad Industrial de Santander https://orcid.org/0000-0002-5465-4769
Camilo Esteban Carrillo Valera Universidad Industrial de Santander https://orcid.org/0009-0004-7848-5009
Jhon Héctor Sandoval Manrique Universidad Industrial de Santander https://orcid.org/0009-0008-1895-9398
Andrés Felipe Luna Chacón Universidad Industrial de Santander https://orcid.org/0009-0005-4273-703X
Iván David Serna Suárez Universidad Industrial de Santander https://orcid.org/0000-0003-4079-2252

DOI:

https://doi.org/10.14482/inde.44.01.103.911

Keywords:

Distribution systems, microgrids, reinforcement learning, retail electricity pricing

Abstract

This paper presents a retail electricity pricing strategy for multi-microgrid distribution systems based on a policy-driven reinforcement learning algorithm. Developed from the perspective of the distribution system operator (DSO), the approach enables the practical derivation of a set of hourly electricity prices that simultaneously maximize profit from energy exchanges and minimize the system’s peak-to-average load ratio, thereby flattening the aggregate load profile. To address the absence of a complete system model from the DSO's viewpoint, the training process employs a Monte Carlo-based method that generates synthetic data from representative base profiles, enabling extensive interaction between the DSO and its environment. Simulations are conducted to validate the effectiveness of the proposed method. Additionally, a sensitivity analysis is presented to evaluate the influence of key parameters on the training performance and the strategy's effectiveness.

References

Ton, D.T., Smith, M.A., 2012. The U.S. department of energy’s microgrid initiative. The Electricity Journal 25, 84–94. doi:https://doi.org/10.1016/j.tej.2012.09.013.

Farrokhabadi, M., Cañizares, C.A., Simpson-Porco, J.W., Nasr, E., Fan, L., Mendoza-Araya, P.A., Tonkoski, R., Tamrakar, U., Hatziargyriou, N., Lagos, D., Wies, R.W., Paolone, M., Liserre, M., Meegahapola, L., Kabalan, M., Hajimiragha, A.H., Peralta, D., Elizondo, M.A., Schneider, K.P., Tuffner, F.K., Reilly, J., 2020. Microgrid stability definitions, analysis, and examples. IEEE Trans. Power Systems 35, 13–29. doi:10.1109/TPWRS.2019.2925703.

Olivares, D.E., Mehrizi-Sani, A., Etemadi, A.H., Cañizares, C.A., Iravani, R., Kazerani, M., Hajimiragha, A.H., Gomis-Bellmunt, O., Saeedifard, M., Palma-Behnke, R., Jiménez-Estévez, G.A., Hatziargyriou, N.D., 2014. Trends in microgrid control. IEEE Trans. Smart Grid 5, 1905– 1919. doi:10.1109/TSG.2013.2295514.

Asmus, P., 2014. Why microgrids are moving into the mainstream: Improving the efficiency of the larger power grid. IEEE Electr. Mag. 2, 12–19. doi:10.1109/MELE.2013.2297021.

Kumar Nunna, H.S.V.S., Doolla, S., 2013. Multiagent-based distributed-energy-resource man- agement for intelligent microgrids. IEEE Trans. Industrial Electronics 60, 1678–1687. doi:10.1109/TIE.2012.2193857.

Mao, M., Jin, P., Hatziargyriou, N.D., Chang, L., 2014. Multiagent-based hybrid energy management system for microgrids. IEEE Trans. Sustainable Energy 5, 938–946. doi:10.1109/TSTE.2014.2313882.

Arefifar, S.A., Ordonez, M., Mohamed, Y.A.R.I., 2017. Energy management in multi-microgrid systems—development and assessment. IEEE Trans. on Power Systems 32, 910–922. doi:10.1109/TPWRS.2016.2568858.

Jiang, W., Yang, K., Yang, J., Mao, R., Xue, N., Zhuo, Z., 2019. A multiagent-based hierarchical energy management strategy for maximization of renewable energy consumption in interconnected multi-microgrids. IEEE Access 7, 169931–169945. doi:10.1109/ACCESS.2019.2955552.

Yang, J., Zhao, J., Luo, F., Wen, F., Dong, Z.Y., 2018b. Decision-making for electricity retailers: A brief survey. IEEE Trans. Smart Grid 9, 4140–4153. doi:10.1109/TSG.2017.2651499.

Braithwait, S.D., 2018. Retail pricing responses to the challenge of distributed energy resources. The Electricity Journal 31, 38–43. doi:https://doi.org/10.1016/j.tej.2018.09.001.

Wang, B., Guo, Q., Yu, Y., 2022. Mechanism design for data sharing: An electricity retail perspective. Applied Energy 314, 118871. doi:https://doi.org/10.1016/j.apenergy.2022.118871.

Thirugnanam, K., Moursi, M.S.E., Khadkikar, V., Zeineldin, H.H., Al Hosani, M., 2021. Energy management of grid interconnected multi-microgrids based on p2p energy ex- change: A data driven approach. IEEE Trans. Power Systems 36, 1546–1562. doi:10.1109/TPWRS.2020.3025113.

Qiu, D., Wang, Y., Wang, J., Jiang, C., Strbac, G., 2023. Personalized retail pricing design for smart metering consumers in electricity market. Applied Energy 348, 121545. doi:https://doi.org/10.1016/j.apenergy.2023.121545.

Trujillo-Baute, E., del Río, P., Mir-Artigues, P., 2018. Analysing the impact of re- newable energy regulation on retail electricity prices. Energy Policy 114, 153–164. doi:https://doi.org/10.1016/j.enpol.2017.11.042.

Yang, H., Zhang, J., Qiu, J., Zhang, S., Lai, M., Dong, Z.Y., 2018a. A practical pricing approach to smart grid demand response based on load classification. IEEE Trans. Smart Grid 9, 179–190. doi:10.1109/TSG.2016.2547883.

Liu, W., Zhuang, P., Liang, H., Peng, J., Huang, Z., 2018. Distributed economic dispatch in microgrids based on cooperative reinforcement learning. IEEE Trans. Neural Networks and Learning Systems 29, 2192–2203. doi:10.1109/TNNLS.2018.2801880.

Shuai, H., He, H., 2021. Online scheduling of a residential microgrid via monte-carlo tree search and a learned model. IEEE Trans. Smart Grid 12, 1073–1087. doi:10.1109/TSG.2020.3035127.

Kuznetsova, E., Li, Y.F., Ruiz, C., Zio, E., Ault, G., Bell, K., 2013. Reinforcement learning for microgrid energy management. Energy 59, 133–146. doi:https://doi.org/10.1016/j.energy.2013.05.060.

Ye, Y., Qiu, D., Wu, X., Strbac, G., Ward, J., 2020. Model-free real-time autonomous control for a residential multi-energy system using deep reinforcement learning. IEEE Trans. Smart Grid 11, 3068–3082. doi:10.1109/TSG.2020.2976771.

Zhang, Q., Dehghanpour, K., Wang, Z., Huang, Q., 2020. A learning-based power management method for networked microgrids under incomplete information. IEEE Trans. Smart Grid 11, 1193–1204. doi:10.1109/TSG.2019.2933502.

Lee, S., Choi, D.H., 2021. Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learn- ing approach. Applied Energy 304, 117754. doi:https://doi.org/10.1016/j.apenergy.2021.117754.

Ghanbari, S., Bahramara, S., Golpîra, H., 2024. Modeling market trading strategies of the intermediary entity for microgrids: A reinforcement learning-based approach. Electric Power Systems Research 227, 109989. doi:https://doi.org/10.1016/j.epsr.2023.109989.

Zhang, Y., Yang, Q., Li, D., An, D., 2022. A reinforcement and imitation learning method for pricing strategy of electricity retailer with customers’ flexibility. Applied Energy 323, 119543. doi:https://doi.org/10.1016/j.apenergy.2022.119543.

Kim, B.G., Zhang, Y., van der Schaar, M., Lee, J.W., 2016. Dynamic pricing and energy con- sumption scheduling with reinforcement learning. IEEE Trans. Smart Grid 7, 2187–2198. doi:10.1109/TSG.2015.2495145.

Lu, R., Hong, S.H., Zhang, X., 2018. A dynamic pricing demand response algo- rithm for smart grid: Reinforcement learning approach. Applied Energy 220, 220–230. doi:https://doi.org/10.1016/j.apenergy.2018.03.072.

Xiong, L., Tang, Y., Mao, S., Liu, H., Meng, K., Dong, Z., Qian, F., 2022. A two-level en- ergy management strategy for multi-microgrid systems with interval prediction and reinforce- ment learning. IEEE Trans. Circuits and Systems I: Regular Papers 69, 1788–1799. doi:10.1109/TCSI.2022.3141229.

Guo, C., Wang, X., Zheng, Y., Zhang, F., 2021. Optimal energy management of multi-microgrids connected to distribution system based on deep reinforcement learning. International Journal of Electrical Power Energy Systems 131, 107048. doi:https://doi.org/10.1016/j.ijepes.2021.107048.

Du, Y., Li, F., 2020. Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning. IEEE Trans. Smart Grid 11, 1066– 1076. doi:10.1109/TSG.2019.2930299.

Peng, D., Xiao, H., Pei, W., Kong, L., 2021. Interactive pricing optimization of multi-microgrid based on deep learning, in: 2021 IEEE 1st Int. Conf. on Digital Twins and Parallel Intelligence (DTPI), pp. 82–85. doi:10.1109/DTPI52967.2021.9540113.

Bukar, A.L., Tan, C.W., Lau, K.Y., 2019. Optimal sizing of an autonomous photo- voltaic/wind/battery/diesel generator microgrid using grasshopper optimization algorithm. Solar Energy 188, 685–696. doi:https://doi.org/10.1016/j.solener.2019.06.050.

Jiménez-Vargas, I., Rey, J.M., Osma-Pinto, G., 2023. Sizing of hybrid microgrids considering life cycle assessment. Renewable Energy 202, 554–565. doi:10.1016/j.renene.2022.11.103.

Lujano-Rojas, J.M., Monteiro, C., Dufo-López, R., Bernal-Agustín, J.L., 2012. Optimum load management strategy for wind/diesel/battery hybrid power systems. Renewable Energy 44, 288– 295. doi:https://doi.org/10.1016/j.renene.2012.01.097.

Rey, J.M., Jiménez-Vargas, I., Vergara, P.P., Osma-Pinto, G., Solano, J., 2022. Sizing of an autonomous microgrid considering droop control. International Journal of Electrical Power Energy Systems 136, 107634. doi:https://doi.org/10.1016/j.ijepes.2021.107634.

Ayop, R., Tan, C.W., 2017. A comprehensive review on photovoltaic emulator. Ren. and Sustainable Energy Reviews 80, 430–452. doi:https://doi.org/10.1016/j.rser.2017.05.217.

Rey, J.M., Vergara, P.P., Solano, J., Ordóñez, G., 2019. Design and optimal sizing of microgrids, in: Microgrids Design and Implementation. Springer, pp. 337–367.