Polygroups are a generalization of groups in which the composition of any two elements are a non-empty set. In this paper, first we recall the concept of polygroups and introduce a new construction for building a polygroup from a polygroup and a non-empty set. Then we study the concept of generalized Cayley graphs over polygroups, say GCP-graphs. Then we prove some properties of them in order to answer this question: which simple graphs are GCP-graphs? Finally, we prove that every simple graph of order at most five is a GCP-graph.